\section{Introduction}
\label{sec:introduction}
A plethora of observations have led to confirm the standard $\Lambda$CDM framework as the most economical and successful model describing our current universe.
This simple picture (pressureless dark matter, baryons and a cosmological constant representing the vacuum energy) has been shown to provide an excellent fit to cosmological data.
However, there are a number of inconsistencies that persist and, instead of diluting with improved precision measurements, gain significance~\cite{Freedman:2017yms,DiValentino:2020zio,DiValentino:2020vvd,DiValentino:2020srs,Freedman:2021ahq,DiValentino:2021izs,Schoneberg:2021qvd,Nunes:2021ipq,Perivolaropoulos:2021jda,Shah:2021onj}.

The most exciting (i.e.\ probably non due to systematics) and most statistically significant ($4-6\sigma$) tension in the literature is the so-called Hubble constant tension, which refers to the discrepancy between cosmological predictions and low redshift estimates of $H_0$~\cite{Verde:2019ivm,Riess:2019qba,DiValentino:2020vnx}.
Within the $\Lambda$CDM scenario, Cosmic Microwave Background (CMB) measurements from the Planck satellite provide a value of $H_0=67.36\pm 0.54$~km s$^{-1}$ Mpc$^{-1}$ at 68\%~CL~\cite{Planck:2018vyg}.
Near universe, local measurements of $H_0$, using the cosmic distance ladder calibration of Type Ia Supernovae with Cepheids, as those carried out by the SH0ES team, provide a measurement of the Hubble constant $H_0=73.2\pm 1.3$~km s$^{-1}$ Mpc$^{-1}$ at 68$\%$~CL~\cite{Riess:2020fzl}.
This problematic $\sim 4\sigma$ discrepancy aggravates when considering other late-time estimates of $H_0$.
For instance, measurements from the Megamaser Cosmology Project~\cite{Pesce:2020xfe}, or those exploiting Surface Brightness Fluctuations~\cite{Blakeslee:2021rqi} only exacerbate this tension~\footnote{%
Other estimates are unable to disentangle between nearby universe and CMB measurements. These include results from the Tip of the Red Giant Branch~\cite{Freedman:2021ahq},
from the astrophysical strong lensing observations~\cite{Birrer:2020tax}
or from gravitational wave events~\cite{Abbott:2017xzu}.}.

As previously mentioned, the SH0ES collaboration exploits the cosmic distance ladder calibration of Type Ia Supernovae, which means that these observations do not provide a direct extraction of the Hubble parameter.
More concretely, the SH0ES team measures the absolute peak magnitude $M_B$ of Type Ia Supernovae \emph{standard candles} and then translates these measurements into an estimate of $H_0$ by means of the magnitude-redshift relation of the Pantheon Type Ia Supernovae sample~\cite{Scolnic:2017caz}.
Therefore, strictly speaking, the SH0ES team does not directly extract the value of $H_0$, and there have been arguments in the literature aiming to translate the Hubble constant tension into a Type Ia Supernovae absolute magnitude tension $M_B$~\cite{Camarena:2019rmj,Efstathiou:2021ocp,Camarena:2021jlr}.
In this regard, late-time exotic cosmologies have been questioned as possible solutions to the Hubble constant tension~\cite{Efstathiou:2021ocp,Camarena:2021jlr}, since within these scenarios, it is possible that the supernova absolute magnitude $M_B$ used to derive the low redshift estimate of $H_0$ is no longer compatible with the $M_B$ needed to fit supernovae, BAO and CMB data.

A number of studies have prescribed to use in the statistical analyses a prior on the intrinsic magnitude rather than on the Hubble constant $H_0$~\cite{Camarena:2021jlr,Schoneberg:2021qvd}.
Following the very same logic of these previous analyses, we reassess here the potential of interacting dark matter-dark energy cosmology~\cite{Amendola:1999er}
in resolving the Hubble constant (\cite{Kumar:2016zpg, Murgia:2016ccp, Kumar:2017dnp, DiValentino:2017iww, Yang:2018ubt, Yang:2018euj, Yang:2019uzo, Kumar:2019wfs, Pan:2019gop, Pan:2019jqh, DiValentino:2019ffd, DiValentino:2019jae, DiValentino:2020leo, DiValentino:2020kpf, Gomez-Valent:2020mqn, Yang:2019uog, Lucca:2020zjb, Martinelli:2019dau, Yang:2020uga, Yao:2020hkw, Pan:2020bur, DiValentino:2020vnx, Yao:2020pji, Amirhashchi:2020qep, Yang:2021hxg, Gao:2021xnk, Lucca:2021dxo, Kumar:2021eev,Yang:2021oxc,Lucca:2021eqy,Halder:2021jiv}
and references therein)
and/or the intrinsic magnitude $M_B$ tension, by demonstrating explicitly from a full analysis that the results are completely independent of whether a prior on $M_B$ or $H_0$ is assumed (see also the recent~\cite{Nunes:2021zzi}).

\section{Theoretical framework}
\label{sec:theory}
We adopt a flat cosmological model described by the Friedmann-Lema\^{i}tre-Robertson-Walker metric.
A possible parameterization of a dark matter-dark energy interaction is provided by the following expressions~\cite{Valiviita:2008iv,Gavela:2009cy}:

\begin{eqnarray}
  \label{eq:conservDM}
\nabla_\mu T^\mu_{(dm)\nu} &=& Q \,u_{\nu}^{(dm)}/a~, \\
  \label{eq:conservDE}
\nabla_\mu T^\mu_{(de)\nu} &=&-Q \,u_{\nu}^{(dm)}/a~.
\end{eqnarray}
In the equations above, $T^\mu_{(dm)\nu}$ and $T^\mu_{(de)\nu}$ represent the energy-momentum tensors for the dark matter and dark energy components respectively, the function $Q$ is the interaction rate between the two dark components, and $u_{\nu}^{(dm)}$ represents the dark matter four-velocity.
In what follows we shall restrict ourselves to the case in which  the
interaction rate is proportional to the dark energy density $\rho_{de}$~\cite{Valiviita:2008iv,Gavela:2009cy}:
\begin{equation}
Q=\ensuremath{\delta{}_{DMDE}}\mathcal{H} \rho_{de}~,
\label{rate}
\end{equation}
where $\ensuremath{\delta{}_{DMDE}}$ is a dimensionless coupling parameter and
$\mathcal{H}=\dot{a}/a$~\footnote{The dot indicates derivative respect to conformal time $d\tau=dt/a$.}.
The background evolution equations in the coupled model considered
here read~\cite{Gavela:2010tm}
\begin{eqnarray}
\label{eq:backDM}
\dot{{\rho}}_{dm}+3{\mathcal H}{\rho}_{dm}
&=&
\ensuremath{\delta{}_{DMDE}}{\mathcal H}{\rho}_{de}~,
\\
\label{eq:backDE}
\dot{{\rho}}_{de}+3{\mathcal H}(1+\ensuremath{w_{\rm 0,fld}}){\rho}_{de}
&=&
-\ensuremath{\delta{}_{DMDE}}{\mathcal H}{\rho}_{de}~.
\end{eqnarray}
The evolution of the dark matter and dark energy density perturbations and velocities divergence field are described in \cite{DiValentino:2019jae} and references therein.

It has been shown in the literature that this model is free of instabilities
if the sign of the  coupling $\ensuremath{\delta{}_{DMDE}}$ and the sign of $(1+\ensuremath{w_{\rm 0,fld}})$ are opposite,
where $\ensuremath{w_{\rm 0,fld}}$ refers to the dark energy equation of state~\cite{He:2008si,Gavela:2009cy}.
In order to satisfy such stability conditions, we explore three possible scenarios, all of them with a redshift-independent equation of state.
In Model A,  the equation of state $\ensuremath{w_{\rm 0,fld}}$ is fixed to $-0.999$.
Consequently, since $(1+\ensuremath{w_{\rm 0,fld}}) >0$, in order to ensure a instability-free perturbation evolution, the dark matter-dark energy coupling $\ensuremath{\delta{}_{DMDE}}$ is allowed to vary in a negative range.
In Model B, $\ensuremath{w_{\rm 0,fld}}$ is allowed to vary but we ensure that the condition $(1+\ensuremath{w_{\rm 0,fld}})>0$ is always satisfied.
Therefore, the coupling parameter  $\ensuremath{\delta{}_{DMDE}}$ is also negative.
In Model C, instead, the dark energy equation of state is phantom ($\ensuremath{w_{\rm 0,fld}}<-1$), therefore the dark matter-dark energy coupling is taken as positive to avoid early-time instabilities.
We shall present separately the cosmological constraints for these three models, together with those corresponding to the canonical $\Lambda$CDM.

\begin{table}[t]
    \centering
    \begin{tabular}{c|c|c}
    Model     & Prior $\ensuremath{w_{\rm 0,fld}}$ & Prior $\ensuremath{\delta{}_{DMDE}}$ \\
    \hline
    A     & -0.999 & [-1.0, 0.0]\\
    B     & [-0.999, -0.333] & [-1.0, 0.0] \\
    C     & [-3, -1.001]& [0.0, 1.0] \\
    \end{tabular}
    \caption{Priors of $\ensuremath{w_{\rm 0,fld}}$, $\delta$ in models A, B, C.}
    \label{tab:priors}
\end{table}

\section{Datasets and Methodology}
\label{sec:data}

In this Section, we present the data sets and methodology employed to obtain the observational constraints on the model parameters by performing Bayesian Monte Carlo Markov Chain (MCMC) analyses.
In order to constrain the parameters, we use the following data sets:
\begin{itemize}
\item The Cosmic Microwave Background (CMB) temperature and polarization power spectra from the final release of Planck 2018, in particular we adopt the plikTTTEEE+lowl+lowE likelihood \cite{Aghanim:2018eyx,Aghanim:2019ame}, plus the CMB lensing reconstruction from the four-point correlation function~\cite{Aghanim:2018oex}.
\item Type Ia Supernovae distance moduli measurements from the \textit{Pantheon} sample~\cite{Scolnic:2017caz}. These measurements constrain the uncalibrated luminosity distance $H_0d_L(z)$, or in other words the slope of the late-time expansion rate (which in turn constrains the current matter energy density, $\Omega_{\rm 0,m}$). We refer to this dataset as \textit{SN}.
\item Baryon Acoustic Oscillations (BAO) distance and expansion rate measurements from the 6dFGS~\cite{Beutler:2011hx}, SDSS-DR7 MGS~\cite{Ross:2014qpa}, BOSS DR12~\cite{Alam:2016hwk} galaxy surveys,
as well as from the eBOSS DR14 Lyman-$\alpha$ (Ly$\alpha$) absorption~\cite{Agathe:2019vsu} and Ly$\alpha$-quasars cross-correlation~\cite{Blomqvist:2019rah}.
These consist of isotropic BAO measurements of $D_V(z)/r_d$
(with $D_V(z)$ and $r_d$ the spherically averaged volume distance and sound horizon at baryon drag, respectively)
for 6dFGS and MGS, and anisotropic BAO measurements of $D_M(z)/r_d$ and $D_H(z)/r_d$
(with $D_M(z)$ the comoving angular diameter distance and $D_H(z)=c/H(z)$ the radial distance)
for BOSS DR12, eBOSS DR14 Ly$\alpha$, and eBOSS DR14 Ly$\alpha$-quasars cross-correlation.
\item A gaussian prior on $M_B= -19.244 \pm 0.037$~mag~\cite{Camarena:2021jlr}, corresponding to the SN measurements from SH0ES.
\item A gaussian prior on the Hubble constant $H_0=73.2\pm 1.3$~km s$^{-1}$ Mpc$^{-1}$ in
agreement with the measurement obtained by the
SH0ES collaboration in~\cite{Riess:2020fzl}.
\end{itemize}
For the sake of brevity, data combinations are indicated as CMB+SN+BAO (CSB), CMB+SN+BAO+$H_0$ (CSBH) and CMB+SN+BAO+$M_B$ (CSBM).

Cosmological observables are computed with \texttt{CLASS}~\cite{Blas:2011rf,Lesgourgues:2011re}.
In order to derive bounds on the proposed scenarios, we modify the efficient and well-known cosmological package \texttt{MontePython}~\cite{Brinckmann:2018cvx}, supporting the Planck 2018 likelihood~\cite{Planck:2019nip}.
We make use of CalPriorSNIa, a module for \texttt{MontePython}, publicly available at \url{https://github.com/valerio-marra/CalPriorSNIa}, that implements an effective calibration prior on the absolute magnitude of Type Ia Supernovae~\cite{Camarena:2019moy,Camarena:2021jlr}.

\section{Main results and discussion}
\label{sec:results}

\begin{figure*}[t]
\begin{center}
\includegraphics[width=0.7\textwidth]{H0.pdf}
\caption{Posterior distribution of the Hubble parameter in the $\Lambda$CDM model (black) and in interacting cosmologies, with priors on the parameters as given in Tab.~\ref{tab:priors}.
We show constraint obtained within model A (green), model B (red) and model C (blue)
for the CMB+SN+BAO data combination (solid lines),
CMB+SN+BAO+$H_0$ (dashed lines)
and CMB+SN+BAO+$M_B$ (dotted lines).}
\label{fig:h0}
\end{center}
\end{figure*}

\begin{figure*}
\begin{center}
\includegraphics[width=\textwidth]{0_PlSB-vs-0_PlSBH-vs-0_PlSBM_triangle.pdf}
\caption{68\% CL and 95\% CL allowed contours and one-dimensional posterior probabilities on a selection of cosmological parameters within the canonical $\Lambda$CDM picture, considering three data combinations: CMB+SN+BAO (red), CMB+SN+BAO+$H_0$ (blue) and CMB+SN+BAO+$M_B$ (green).}
\label{fig:triangle_LCDM}
\end{center}
\end{figure*}

\begin{table}[t]
\centering
\begin{tabular}{|l|c|c|c|}
\hline
Parameter            & CSB            & CSBH & CSBM \\
\hline
$\omega{}_{cdm }$    & $0.1193\pm0.0010$ & $0.1183\pm0.0009$ & $0.1183_{-0.0009}^{+0.0008}$  \\
$\ensuremath{\Omega_{\rm 0,fld}}$ & $0.6889_{-0.0061}^{+0.0057}$   & $0.6958_{-0.0050}^{+0.0056}$    & $0.6956_{-0.0049}^{+0.0057}$    \\
$\Omega_{\rm 0,m}$     & $0.3111_{-0.0057}^{+0.0061}$   & $0.3042_{-0.0056}^{+0.0050}$    & $0.3044_{-0.0057}^{+0.0049}$    \\
$M_B$                & $-19.42\pm0.01$     & $-19.40\pm0.01$      & $-19.40\pm0.01$       \\
$H_0$                & $67.68_{-0.46}^{+0.41}$        & $68.21_{-0.41}^{+0.42}$        & $68.20_{-0.41}^{+0.41}$         \\
$\sigma_8$           & $0.8108_{-0.0058}^{+0.0061}$   & $0.8092_{-0.0065}^{+0.0060}$    & $0.8090_{-0.0059}^{+0.0064}$     \\
\hline
minimum $\chi^2$ & $3819.46$ & $3836.50$ & $3840.44$ \\
\hline
\end{tabular}
\caption{Mean values and 68\% CL errors on $\omega_{cdm }\equiv\Omega_{cdm} h^2$, the current dark energy density $\ensuremath{\Omega_{\rm 0,fld}}$, the current matter energy density $\Omega_{\rm 0,m}$, the Supernovae Ia intrinsic magnitude $M_B$, the Hubble constant $H_0$ and the clustering parameter $\sigma_8$ within the standard $\Lambda$CDM paradigm. We also report the minimum value of the $\chi^2$ function obtained for each of the data combinations.}
\label{tab:model_LCDM}
\end{table}

We start by discussing the results obtained within the canonical $\Lambda$CDM scenario. Table~\ref{tab:model_LCDM} presents the mean values and the $1\sigma$ errors on a number of different cosmological parameters.
Namely, we show the constraints on
$\omega_{cdm }\equiv\Omega_{0,cdm} h^2$,
the current dark energy density $\ensuremath{\Omega_{\rm 0,fld}}$,
the current matter energy density $\Omega_{\rm 0,m}$,
the Supernovae Ia intrinsic magnitude $M_B$,
the Hubble constant $H_0$ and the clustering parameter $\sigma_8$
arising from three possible data combinations considered here and above described:
CMB+SN+BAO (CSB), CMB+SN+BAO+$H_0$ (CSBH), CMB+SN+BAO+$M_B$ (CSBM).
Interestingly, \emph{all} the parameters experience the very same shift regardless the prior is adopted on the Hubble constant or on the intrinsic Supernovae Ia magnitude $M_B$.
The mean value of $H_0$ coincides for both the CSBH and the CSBM data combinations, as one can clearly see from the dashed and dotted black lines in Fig.~\ref{fig:h0}.
Figure~\ref{fig:triangle_LCDM} presents the two-dimensional allowed  contours and the one-dimensional posterior probabilities on the parameters shown in Tab.~\ref{tab:model_LCDM}.
Notice that all the parameters are equally shifted when adding the prior on $H_0$ or on $M_B$, except for $\sigma_8$ which remains almost unchanged. Notice also that the value of the current matter density, $\Omega_{\rm 0,m}$, is smaller when a prior from SN measurements is considered:
due to the larger $H_0$ value that these measurements imply, in order to keep the CMB peaks structure unaltered, the value of $\Omega_{\rm 0,m}$ should be smaller to ensure that the product $\omega_m h^2$ is barely shifted.

\begin{table}[t]
\centering
\begin{tabular}{|l|c|c|c|}
\hline
Parameter            & CSB            & CSBH & CSBM \\
\hline
$\omega{}_{cdm }$    & $0.107_{-0.005}^{+0.011}$ & $0.09\pm0.01$ & $0.096_{-0.009}^{+0.011}$  \\
$\ensuremath{\Omega_{\rm 0,fld}}$ & $0.723_{-0.028}^{+0.017}$ & $0.758_{-0.024}^{+0.026}$    & $0.754_{-0.028}^{+0.025}$     \\
$\Omega_{\rm 0,m}$   & $0.277_{-0.017}^{+0.028}$ & $0.242_{-0.026}^{+0.024}$    & $0.246_{-0.025}^{+0.028}$     \\
$\ensuremath{\delta{}_{DMDE}}$         & $-0.116_{-0.044}^{+0.100}$  & $-0.219_{-0.086}^{+0.083}$   & $-0.203_{-0.087}^{+0.093}$    \\
$M_B$                & $-19.40\pm0.02$  & $-19.38_{-0.01}^{+0.02}$    & $-19.37\pm0.02$    \\
$H_0$                & $68.59_{-0.79}^{+0.65}$    & $69.73_{-0.72}^{+0.71}$       & $69.67_{-0.85}^{+0.75}$       \\
$\sigma_8$           & $0.90_{-0.08}^{+0.04}$  & $1.01_{-0.11}^{+0.08}$      & $1.00_{-0.12}^{+0.07}$      \\
\hline
minimum $\chi^2$ & $3819.86$ & $3831.90$ & $3835.86$ \\
\hline
\end{tabular}
\caption{Mean values and 68\% CL errors on $\omega_{cdm }\equiv\Omega_{cdm} h^2$, the current dark energy density $\ensuremath{\Omega_{\rm 0,fld}}$, the current matter energy density $\Omega_{\rm 0,m}$, the dimensionless dark matter-dark energy coupling $\ensuremath{\delta{}_{DMDE}}$, the Supernovae Ia intrinsic magnitude $M_B$, the Hubble constant $H_0$ and the clustering parameter $\sigma_8$ within the interacting model A, see Tab.~\ref{tab:priors}. We also report the minimum value of the $\chi^2$ function obtained for each of the data combinations.}
\label{tab:model_A}
\end{table}

\begin{figure*}
\begin{center}
\includegraphics[width=\textwidth]{A_PlSB-vs-A_PlSBH-vs-A_PlSBM_triangle.pdf}
\caption{68\% CL and 95\% CL allowed contours and one-dimensional posterior probabilities on a selection of cosmological parameters within model A, considering three data combinations: CMB+SN+BAO (red), CMB+SN+BAO+$H_0$ (blue) and CMB+SN+BAO+$M_B$ (green).}
\label{fig:triangle_A}
\end{center}
\end{figure*}

We focus now on Model A, which refers to an interacting cosmology with $\ensuremath{w_{\rm 0,fld}}=-0.999$ and $\ensuremath{\delta{}_{DMDE}}<0$.
Table~\ref{tab:model_A} presents the mean values and the $1\sigma$ errors  on the same cosmological parameters listed above, with the addition of the coupling parameter $\ensuremath{\delta{}_{DMDE}}$, for the same three data combination already discussed.

Notice again that all the parameters are equally shifted to either smaller or larger values, regardless the prior is adopted on either $H_0$ or $M_B$. In this case the shift on the Hubble parameter is larger than that observed within the $\Lambda$CDM model, as one can notice from the blue curves depicted in
Fig.~\ref{fig:h0}.
Interestingly, we observe a $2\sigma$ indication in favor of a non-zero value of the coupling $\ensuremath{\delta{}_{DMDE}}$ when considering the CSBH and the CSBM data combinations.
Indeed, while the value of the minimum $\chi^2$ is almost equal to that obtained in the $\Lambda$CDM framework for the CSB data analyses, when adding either a prior on $H_0$ or on $M_B$,
the minimum $\chi^2$ value is \emph{smaller} than that obtained for the standard cosmological picture: therefore, the addition of a coupling \emph{improves} the overall fit.
Figure~\ref{fig:triangle_A} presents the two-dimensional allowed contours and the one-dimensional posterior probabilities obtained within Model A.
It can be noticed that the prior on the Hubble constant and on the intrinsic magnitude lead to the very same shift, and the main conclusion is therefore prior-independent:
there is a $\sim 2\sigma$ indication for a non-zero dark matter-dark energy coupling when considering either $H_0$ or $M_B$ measurements,
\emph{and} the value of the Hubble constant is considerably larger, alleviating the $H_0$ tension.

\begin{table}[t]
\centering
\begin{tabular}{|l|c|c|c|}
\hline
Parameter            & CSB            & CSBH & CSBM \\
\hline
$\omega{}_{cdm }$    & $0.077_{-0.014}^{+0.036}$  & $0.061_{-0.019}^{+0.034}$  & $0.065_{-0.017}^{+0.036}$  \\
$\ensuremath{\Omega_{\rm 0,fld}}$ & $0.785_{-0.081}^{+0.034}$  & $0.825_{-0.070}^{+0.045}$  & $0.818_{-0.075}^{+0.041}$  \\
$\Omega_{\rm 0,m}$   & $0.215_{-0.034}^{+0.081}$  & $0.174_{-0.044}^{+0.069}$  & $0.182_{-0.041}^{+0.075}$  \\
$\ensuremath{w_{\rm 0,fld}}$      & $-0.909_{-0.090}^{+0.026}$ & $-0.917_{-0.082}^{+0.026}$ & $-0.918_{-0.081}^{+0.026}$  \\
$\ensuremath{\delta{}_{DMDE}}$         & $-0.35_{-0.14}^{+0.26}$    & $-0.45_{-0.16}^{+0.22}$    & $-0.43_{-0.15}^{+0.24}$   \\
$M_B$                & $-19.41\pm0.02$            & $-19.38\pm0.02$            & $-19.38\pm0.02$  \\
$H_0$                & $68.28_{-0.85}^{+0.79}$    & $69.68_{-0.75}^{+0.71}$    & $69.57_{-0.76}^{+0.75}$     \\
$\sigma_8$           & $1.30_{-0.51}^{+0.01}$     & $1.60_{-0.76}^{+0.06}$     & $1.53_{-0.71}^{+0.03}$    \\
\hline
minimum $\chi^2$ & $ 3819.96$ & $3832.28$ & $3836.24$ \\
\hline
\end{tabular}
\caption{Mean values and 68\% CL errors on $\omega_{cdm }\equiv\Omega_{cdm} h^2$, the current dark energy density $\ensuremath{\Omega_{\rm 0,fld}}$, the current matter energy density $\Omega_{\rm 0,m}$, the dark energy equation of state $\ensuremath{w_{\rm 0,fld}}$,
the dimensionless dark matter-dark energy coupling $\ensuremath{\delta{}_{DMDE}}$, the Supernovae Ia intrinsic magnitude $M_B$, the Hubble constant $H_0$ and the clustering parameter $\sigma_8$ within the interacting model B, see Tab.~\ref{tab:priors}.
We also report the minimum value of the $\chi^2$ function obtained for each of the data combinations.}
\label{tab:model_B}
\end{table}

\begin{figure*}
\begin{center}
\includegraphics[width=\textwidth]{B_PlSB-vs-B_PlSBH-vs-B_PlSBM_triangle.pdf}
\caption{68\% CL and 95\% CL allowed contours and one-dimensional posterior probabilities on a selection of cosmological parameters within model B, considering three data combinations: CMB+SN+BAO (red), CMB+SN+BAO+$H_0$ (blue) and CMB+SN+BAO+$M_B$ (green).}
\label{fig:triangle_B}
\end{center}
\end{figure*}

Focusing now on Model B, which assumes a negative coupling $\ensuremath{\delta{}_{DMDE}}$ and a constant, but freely varying, dark energy equation of state $\ensuremath{w_{\rm 0,fld}}$ within the $\ensuremath{w_{\rm 0,fld}}>-1$ region,
we notice again the same shift on the cosmological parameters, regardless the prior is introduced in the Hubble parameter ($H_0$) or in the Supernovae Ia intrinsic magnitude ($M_B$), as can be noticed from Tab.~\ref{tab:model_B}.
As in Model A, the value of $H_0$ in this interacting cosmology is larger than within the $\Lambda$CDM framework (see the red curves in Fig.~\ref{fig:h0}),
albeit slightly smaller than in Model A, due to the strong anti-correlation between $\ensuremath{w_{\rm 0,fld}}$ and $H_0$~\cite{DiValentino:2016hlg,DiValentino:2019jae}.
Consequently, a larger value of $\ensuremath{w_{\rm 0,fld}}>-1$ implies a lower value of $H_0$.
Nevertheless, a $2\sigma$ preference for a non-zero value of the dark matter-dark energy coupling is present also in this case, and also when the CSB dataset is considered:
for the three data combinations presented here, there is always a preference for a non-zero dark matter-dark energy coupling.
Notice that the minimum $\chi^2$ in Model B is smaller than that corresponding to the minimal $\Lambda$CDM framework, but slightly larger than that of Model A, which is nested in Model B. The differences between the minimum $\chi^2$ in Model A and Model B, however, are small
enough to be considered as numerical fluctuations. Since, as previously stated, $\ensuremath{w_{\rm 0,fld}}$ and $H_0$ are strongly anti-correlated, a more negative value of the dark energy equation of state (i.e.\ $\ensuremath{w_{\rm 0,fld}}=-0.999$ as in Model A, close to the prior limit) is preferred by both the CSBH and the CSBM data combinations.

In Fig.~\ref{fig:triangle_B} we depict the two-dimensional allowed contours and the one-dimensional posterior probabilities obtained for Model B.
From a comparison to Fig.~\ref{fig:triangle_LCDM} and also confronting the mean values of Tab.~\ref{tab:model_B} to those shown in Tab.~\ref{tab:model_LCDM} (and, to a minor extent, to those in Tab.~\ref{tab:model_A}),
one can notice that the value of  \ $\ensuremath{\Omega_{\rm 0,fld}}$ is much larger.
The reason for this is related to the lower value for the present matter energy density $\Omega_{\rm 0,m}$ (the values are also shown in the tables), which is required within the interacting cosmologies when the dark matter-dark energy coupling is negative.
In the context of a universe with a negative dark coupling, indeed, there is an energy flow from dark matter to dark energy.
Consequently, the (dark) matter content in the past is higher than in the standard $\Lambda$CDM scenario and the amount of intrinsic (dark) matter needed today is lower, because of the extra contribution from the dark energy sector.
In a flat universe, this translates into a much higher value of $\ensuremath{\Omega_{\rm 0,fld}}$.
On the other hand, a lower value of $\Omega_{m,0}$ requires a larger value of the clustering parameter $\sigma_8$ to be able to satisfy the overall normalization of the matter power spectrum. In any case, we find again that the addition of a prior on either $H_0$ or $M_B$ leads to exactly the very same shift for all the cosmological parameters.
Therefore, Model B also provides an excellent solution to the Hubble constant tension,
although at the expense of a very large $\sigma_8$.

\begin{table}[t]
\centering
\begin{tabular}{|l|c|c|c|}
\hline
Parameter            & CSB            & CSBH & CSBM \\
\hline
$\omega{}_{cdm }$    & $0.138_{-0.015}^{+0.008}$   & $0.137_{-0.016}^{+0.007}$  & $0.135_{-0.013}^{+0.008}$ \\
$\ensuremath{\Omega_{\rm 0,fld}}$ & $0.655_{-0.021}^{+0.032}$   & $0.671_{-0.018}^{+0.031}$ & $0.675_{-0.018}^{+0.027}$  \\
$\Omega_{\rm 0,m}$   & $0.345_{-0.032}^{+0.021}$   & $0.329_{-0.031}^{+0.018}$ & $0.325_{-0.027}^{+0.018}$  \\
$\ensuremath{w_{\rm 0,fld}}$      & $-1.087_{-0.042}^{+0.051}$  & $-1.131_{-0.044}^{+0.053}$ & $-1.117_{-0.044}^{+0.048}$  \\
$\ensuremath{\delta{}_{DMDE}}$         & $0.183_{-0.180}^{+0.061}$   & $0.173_{-0.170}^{+0.051}$  & $0.150_{-0.150}^{+0.051}$   \\
$M_B$                & $-19.41\pm0.02$             & $-19.38\pm0.02$            & $-19.37\pm0.02$ \\
$H_0$                & $68.29_{-0.91}^{+0.66}$     & $69.74_{-0.73}^{+0.75}$    & $69.67_{-0.77}^{+0.78}$     \\
$\sigma_8$           & $0.735_{-0.057}^{+0.045}$   & $0.748_{-0.041}^{+0.068}$  & $0.755_{-0.047}^{+0.051}$  \\
\hline
minimum $\chi^2$ & $3818.24$ & $3830.56$ & $3835.10$ \\
\hline
\end{tabular}
\caption{Mean values and 68\% CL errors on $\omega_{cdm }\equiv\Omega_{cdm} h^2$, the current dark energy density $\ensuremath{\Omega_{\rm 0,fld}}$, the current matter energy density $\Omega_{\rm 0,m}$, the dark energy equation of state $\ensuremath{w_{\rm 0,fld}}$,
the dimensionless dark matter-dark energy coupling $\ensuremath{\delta{}_{DMDE}}$, the Supernovae Ia intrinsic magnitude $M_B$, the Hubble constant $H_0$ and the clustering parameter $\sigma_8$ within the interacting model C, see Tab.~\ref{tab:priors}.
We also report the minimum value of the $\chi^2$ function obtained for each of the data combinations.}
\label{tab:model_C}
\end{table}

\begin{figure*}
\begin{center}
\includegraphics[width=\textwidth]{C_PlSB-vs-C_PlSBH-vs-C_PlSBM_triangle.pdf}
\caption{68\% CL and 95\% CL allowed contours and one-dimensional posterior probabilities on a selection of cosmological parameters within model C, considering three data combinations: CMB+SN+BAO (red), CMB+SN+BAO+$H_0$ (blue) and CMB+SN+BAO+$M_B$ (green).}
\label{fig:triangle_C}
\end{center}
\end{figure*}

Finally, Tab.~\ref{tab:model_C} shows the mean values and the $1\sigma$ errors on the usual cosmological parameters explored along this study, for Model C.
Notice that this model benefits from both its interacting nature and from the fact that $\ensuremath{w_{\rm 0,fld}}<-1$ and $\ensuremath{\delta{}_{DMDE}}>0$.
Both features of the dark energy sector have been shown to be excellent solutions to the Hubble constant problem.
As in the previous cases, the shift in the cosmological parameters induced by the addition of a prior is independent of its nature, i.e.\ it is independent on whether a prior on $H_0$ or $M_B$ is adopted.
Within this model, the value of the Hubble constant is naturally larger than within the $\Lambda$CDM model (see the blue lines in Fig.~\ref{fig:h0}),
regardless of the data sets assumed in the analyses.
Despite its phantom nature, as in this particular case $\ensuremath{w_{\rm 0,fld}}<-1$ to ensure a instability-free evolution of perturbations, Model C provides the \emph{best-fits to any of the data combinations explored here, performing even better than} the minimal $\Lambda$CDM picture,
as one can clearly notice from the last row of Tab.~\ref{tab:model_C}.
This fact makes Model C a very attractive cosmological scenario which can provide a solution for the long-standing $H_0$ tension. We must remember that model C, however, has two degrees of freedom more than the standard $\Lambda$CDM paradigm.
Figure~\ref{fig:triangle_C} illustrates the two-dimensional allowed contours and the one-dimensional posterior probabilities obtained within Model C.
Notice that here the situation is just the opposite one of Model B: the value of $\ensuremath{\Omega_{\rm 0,fld}}$ is much smaller than in standard scenarios,
due to the larger value required for the present matter energy density $\Omega_{\rm 0,m}$ when the dark matter-dark energy coupling $\ensuremath{\delta{}_{DMDE}}>0$ and $\ensuremath{w_{\rm 0,fld}}<-1$.
This larger value of the present matter energy density also implies a lower value for the clustering parameter $\sigma_8$, in contrast to what was required within Model B.

\section{Final Remarks}
\label{sec:conclusions}

In this study we have tried to reassess the ability of interacting dark matter-dark energy cosmologies in alleviating the long-standing and highly significant Hubble constant tension.
Despite the fact that in the past these models have been shown to provide an excellent solution to the discrepancy between local measurements and high redshift, Cosmic Microwave Background estimates of $H_0$, there have been recent works in the literature questioning
their effectiveness, related to a misinterpretation of SH0ES data, which indeed does not directly extract the value of $H_0$.
We have therefore computed the ability of interacting cosmologies of reducing the Hubble tension by means of two possible different priors in the cosmological analyses:
a prior on the Hubble constant and, separately, a prior on Type Ia Supernova absolute magnitude.
We combine these priors with Cosmic Microwave Background (CMB), Type Ia Supernovae (SN) and Baryon Acoustic Oscillation (BAO) measurements,
showing that the constraints on the cosmological parameters are independent of the choice of prior, and that the Hubble constant tension is always alleviated.
This last statement is also prior-independent.
Furthermore, one of the possible interacting cosmologies considered here,
with a phantom nature, provides a better fit than the canonical $\Lambda$CDM framework for all the considered data combinations, but with two extra degrees of freedom.
We therefore conclude that interacting dark-matter dark-energy cosmologies still provide a very attractive and viable theoretical and phenomenological scenario
where to robustly relieve the Hubble constant tension,
 regardless the method one adopts to process SH0ES data.

\begin{acknowledgments}
\noindent
SG acknowledges financial support from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 754496 (project FELLINI).
EDV is supported by a Royal Society Dorothy Hodgkin Research Fellowship.
OM is supported by the Spanish grants PID2020-113644GB-I00, PROMETEO/2019/083 and by the European ITN project HIDDeN (H2020-MSCA-ITN-2019//860881-HIDDeN).
RCN acknowledges financial support from the Funda\c{c}\~{a}o de Amparo \`{a} Pesquisa do Estado de S\~{a}o Paulo (FAPESP, S\~{a}o Paulo Research Foundation) under the project No. 2018/18036-5.
\end{acknowledgments}


\section{Introduction} \label{sec:introduction}  \input{introduction}
\section{Related Work} \label{sec:related_work}  \input{relatedWork}
\section{Model Description} \label{sec:model} \input{modelDescription}
\section{Experiments} \label{sec:experiments} \input{experiments}
\section{Conclusions and Future Work} \label{sec:conclusions} \input{conclusion}

{\small
\textbf{Acknowledgements}
\input{acknowledgements}
}

{\small
\bibliographystyle{ieee}

\subsection{Composable Activities Dataset} \label{subsec:composableActivities}

\subsection{Inference of per-frame annotations.}
\label{subsec:action_annotation}
The hierarchical structure and compositional
properties of our model enable it to output a predicted global activity,
as well as per-frame annotations of predicted atomic actions and poses for each body
region.
It is important to highlight that in the generation of the per-frame annotations, no prior temporal
segmentation of atomic actions is needed. Also, no post-processing of the output is performed. The
proficiency of our model to produce
per-frame annotated data, enabling action detection temporally and
spatially, make our model unique.

Figure \ref{fig:annotation} illustrates
the capability of our model to provide per-frame annotation of the atomic
actions that compose each activity. The accuracy of
the mid-level action prediction can be evaluated as in \cite{Wei2013}.
Specifically, we first obtain segments of the same predicted action in each
sequence, and then compare these segments with ground truth action labels. The
estimated label of the segment is assumed correct if the detected segment is
completely contained in a ground truth segment with the same label, or if the
Jaccard Index considering the segment and the ground truth label is greater
than 0.6. Using these criteria, the accuracy of the mid-level actions is
79.4\%. In many cases, the wrong action prediction is only highly local in time
or space, and the model is still able to correctly predict the activity label
of the sequence. Taking only the correctly predicted videos in terms of global
activity prediction, the accuracy of action labeling reaches 83.3\%. When consider this number, it
is
important to note that not every ground truth action label is accurate: the
videos were hand-labeled by volunteers, so there is a chance for mistakes in
terms of the exact temporal boundaries of the action. In
this sense, in our experiments we observe cases where the predicted
labels showed more accuracte temporal boundaries than the ground
truth.

\begin{figure*}[th]
\begin{center}
\includegraphics[width=0.999\linewidth]{./fig_all_sequences_red.pdf}
\end{center}
\caption{Per-frame predictions of atomic actions for selected activities,
showing 20 frames of each video. Each frame is joined with the predicted action
annotations of left arm, right arm, left leg and right leg. Besides the prediction of the global
activity of the video, our algorithm is able to
correctly predict the atomic actions that compose each activity in each frame,
as well as the body regions that are active during the execution of the action.
Note that in the example video of the activity \emph{Walking while calling with
hands}, the \emph{calling with hands} action is correctly annotated even when
the subject change the waving hand during the execution of the activity.}
\label{fig:annotation}
\end{figure*}

\subsection{Robustness to occlusion and noisy joints.}
Our method is also capable of inferring action and activity labels even if some
joints are not observed. This is a common situation in practice,
as body motions induce temporal self-occlusions of body regions.
Nevertheless, due to the joint estimation of poses, actions, and activities,
our model is able to reduce the effect of this problem. To illustrate this, we
simulate a totally occluded region by fixing its geometry to the position
observed in the first frame.
We select which region to be completely occluded in every sequence using uniform sampling.
In this scenario, the accuracy of our preliminary model in \cite{Lillo2014} drops
by 7.2\%. Using our new SR setup including NI handling, the accuracy only drops
by 4.3\%, showing that the detection of non-informative poses helps the model
to deal with occluded regions. In fact, as we show in Section
\ref{subsec:exp_non_info_handling}, many of truly occluded regions in the
videos are identified using NI handling. In contrast, the drop in performance of
BoW is 12.5\% and HMM 10.3\%: simpler models are less capable of robustly dealing
with occluded regions, since their pose assignments rely only on the descriptor
itself, while in our model the assigned pose depends on the descriptor,
sequences of poses and actions, and the activity evaluated, making inference
more robust. Fig. \ref{fig:occlusions} shows some qualitative results of
occluded regions.

\begin{figure}[tb]
\begin{center}
\includegraphics[width=0.999\linewidth]
{./subject_1_6.pdf} \\
{\footnotesize Right arm occluded} \\
\includegraphics[width=0.999\linewidth]
{./subject_1_23.pdf}\\
{\footnotesize Left leg occluded}  \\
\includegraphics[width=0.999\linewidth]
{./subject_1_8.pdf}\\
{\footnotesize Left arm occluded}\\
\end{center}
\caption{The occluded body regions are depicted in light blue. When an arm or
leg is occluded, our method still provides a good estimation of the underlying actions in each
frame.}
\label{fig:occlusions}
\end{figure}

In terms of noisy joints, we manually add random Gaussian noise to change the
joints 3D location of testing videos, using the SR setup and the GEO descriptor
to isolate the effect of the joints and not mixing the motion descriptor. Figure
\ref{fig:joint_noise} shows the accuracy of testing videos in terms of noise
dispersion $\sigma_{noise}$ measured in inches. For little noise, there is no
much effect in our model accuracy, as expected for the robustness of the
geometric descriptor. However, for more drastic noise added to every joint, the
accuracy drops dramatically. This behavior is expected, since for highly noisy
joints the model can no longer predict well the sequence of actions and poses.

\begin{figure}[tb]
\begin{center}
\includegraphics[width=0.999\linewidth]{./fig_acc_vs_noise.pdf} \\
\end{center}
\caption{Performance of our model in presence of simulated Gaussian noise in
every joint, as a function of $\sigma_{noise}$ measured in inches. When the
noise is less than 3 inches in average, the model performance is not very
affected, while for bigger noise dispersion the model accuracy is drastically
affected. It is important no note that in our simulation, every joint is
affected to noise, while in a real setup, noisy joint estimation tend to occur
more rarely. } \label{fig:joint_noise}
\end{figure}

\subsection{Early activity prediction.}
Our model needs the complete video to make an accurate activity and action
prediction of a query video. In this section, we analyze the number of frames
(as a percentage of a complete activity sequence) needed
to make an accurate activity prediction. Figure \ref{fig:accuracy_reduced_frames}
shows the mean accuracy over the dataset (using leave-one-subject-out
cross-validation) in function of the
percentage of frames used by the classifier to label each video. We note that
considering 30\% of the frames, the classifier performs reasonable predictions,
while 70\% of frames are needed to closely match the
accuracy of using all frames.
\begin{figure}[tb]
\begin{center}
\includegraphics[width=0.999\linewidth]{./fig_acc_vs_frame_reduction.pdf}
\end{center}
\caption{Accuracy of activity recognition versus percentage of frames used in
Composable Activities dataset. In general, 30\% of the frames are needed to
perform reasonable predictions, while 70\% of frames are needed to closely match the
accuracy of using all frames.}
\label{fig:accuracy_reduced_frames}
\end{figure}

\subsection{Failure cases.}

We also study some of the failure cases that we observe during the
experimentation with our model.
Figure \ref{fig:errors} shows some error cases. It is interesting that
the sequences are confusing even for humans when only the skeleton is available
as in the figure. These errors probably will not be surpassed with the model
itself, and will need to use other sources of information like object
detectors, where a cup should be distinguished from a cellphone as in the
third row of Figure \ref{fig:errors}.

\begin{figure}[tb]
\begin{center}
\includegraphics[width=0.999\linewidth]
{./sbj1_1.pdf} \\
{\footnotesize Ground truth: Walking while calling with hands\\
Prediction: Walking while waving hand} \\
\includegraphics[width=0.999\linewidth]
{./sbj4_4.pdf}\\
{\footnotesize Ground truth: Composed activity 1\\
Prediction: Talking on cellphone and drinking}  \\
\includegraphics[width=0.999\linewidth]
{./sbj4_6.pdf}\\
{\footnotesize Ground truth: Waving hand and drinking\\
Prediction: Talking on cellphone and scratching head} \\
\end{center}
\caption{Failure cases. Our algorithm tends to confuse activities that share very similar
body postures.}
\label{fig:errors}
\end{figure}

\begin{comment}
\subsubsection{New activity characterization}
As we mention in previous section, our model using sparse regularization and
non-negative weights on activity ($\alpha$) classifiers and action ($\beta$)
classifiers do not \emph{punish} poses that have no influence in the
activities. For this reason, our model is able to model a new composed activity
just combining the coefficients of two known activities, leaving the rest of
the parameters of the model untouched. We use an heuristic approach to combine
two models: givint two classes $c_1$ and $c_2$, their coefficients for a region
$r$ and action $a$ are $ \alpha^r_{c_1,a}$ and $ \alpha^r_{c_2,a}$
respectively. For a new class $c_{new}$ composed of classes $c_1$ and $c_2$, we
use the mean value of the coefficients \begin{equation}
\alpha^r_{{c_{new},a}} = \frac{(\alpha^r_{c_1,a} + \alpha^r_{c_2,a})}{2}
\end{equation}
 only when the corresponding coefficients for are positive; in other case, we
use the maximum value of the two coefficients. For all subjects of the dataset,
we create all the combinations od two activities, and tested the new model
using three composed videos per subject. The average accuracy of the activity
$16+1$ is 90.2\%, and in average the activities that compose the new activity
drops its accuracy in 12.3\%, showing that we effectively incorporate a new
composed activity to the model at a little cost of getting more confusion over
the original activities. Moreover, the accuracy of action labeling for the new
class is 74.2\%,  similar to the accuracy of the action labeling of the
original model, so we can effectively transfer the learning of atomic action
classifiers to new compositions of activities.

\begin{table}
\begin{tabular}
\hline
Activity group & Accuracy of new class & \\
\hline
Simple & 92.
Complex & 87.2\% & \\
\hline
All & 90.2\% & \\
\end{tabular}
\caption{}
\label{tab:acc_new_class}
\end{table}

\end{comment}

\subsection{Classification of Simple and Isolated Actions}

As a first experiment,
we evaluate the performance of our model on the task of simple and
isolated human action recognition in the  MSR-Action3D dataset
\cite{WanLi2010}.
Although our model is tailored at recognizing complex
actions, this experiment verifies the performance of our model in the
simpler scenario of isolated atomic action classification.

The MSR-Action3D dataset provides pre-trimmed depth videos and estimated body poses
for isolated actors performing actions from 20
categories. We use 557 videos
in a similar setup to
\cite{Wang2012}, where videos from subjects 1, 3, 5, 7, 9 are used for
training and the rest for testing. Table \ref{tab:msr3d} shows that in this
dataset our model achieves classification accuracies comparable to
state-of-the-art methods.

\begin{table}[t]
\footnotesize
\centering
\begin{tabular}{|l|c|}
\hline
\textbf{Algorithm} & \textbf{Accuracy}\\
\hline
Our model &  93.0\% \\
\hline
L. Tao \etal \cite{Tao2015} & 93.6\% \\
C. Wang \etal \cite{Wang2013} &    90.2\% \\
Vemulapalli \etal \cite{Vemulapalli2014} & 89.5\% \\
\hline
\end{tabular}
\caption{\footnotesize
Recognition accuracy in the MSR-Action3D
dataset.}
\label{tab:msr3d}
\end{table}

\subsection{Detection of Concurrent Actions}
Our second experiment evaluates the performance of our model in a concurrent
action recognition setting. In this scenario, the goal is to predict
the temporal localization of actions that may occur concurrently in a long
video. We evaluate this task on the Concurrent Actions dataset \cite{Wei2013},
which
provides 61 RGBD videos and pose estimation data annotated with 12
action categories.
We use a similar evaluation setup as proposed by the authors.
We split the dataset into training and testing sets with a 50\%-50\% ratio.
We evaluate performance by measuring precision-recall: a detected action
is declared as a true positive if its temporal overlap with the ground
truth action interval is larger than 60\% of their union, or if
the detected interval is completely covered by the ground truth annotation.

Our model is tailored at recognizing complex actions that are composed
of atomic components. However, in this scenario, only atomic actions are
provided and no compositions are explicitly defined. Therefore, we apply
a simple preprocessing step: we cluster training videos into groups
by comparing the occurrence of atomic actions within each video.
The resulting groups are used as complex actions labels in the training
videos of this dataset.
At inference time, our model outputs a single labeling per video,
which corresponds to the atomic action labeling that maximizes the energy of
our model.
Since there are no thresholds to adjust, our model produces the single
precision-recall measurement reported in Table \ref{tab:concurrent}.
Our model outperforms the state-of-the-art method in this
dataset at that recall level.

\begin{table}[tb]
\footnotesize
\centering
\begin{tabular}{|l|c|c|}
\hline
\textbf{Algorithm} & \textbf{Precision} & \textbf{Recall}\\
\hline
Our full model &  0.92 & 0.81 \\
\hline
Wei et al. \cite{Wei2013} & 0.85 & 0.81 \\
\hline
\end{tabular}
\caption{
\footnotesize
Recognition accuracy in the Concurrent Actions dataset. }
\label{tab:concurrent}
\end{table}

\subsection{Recognition of Composable Activities}
In this experiment, we evaluate the performance of our model to recognize complex
and composable human actions. In the evaluation, we use the Composable
Activities dataset \cite{Lillo2014},
which provides 693 videos of 14 subjects performing 16 activities.
Each activity is a spatio-temporal composition of atomic actions.
The dataset provides a total of 26 atomic actions that are shared across
activities. We train our model using two levels of supervision during training:
i) spatial annotations that map body regions to the execution of each action are made available
ii) spatial supervision is not available, and therefore the labels $\vec{v}$ to assign spatial regions to actionlets
are treated as latent variables.

Table \ref{tab:composable} summarizes our results. We observe that under both
training conditions, our model achieves comparable performance. This indicates
that our weakly supervised model can recover some of the information
that is missing while performing well at the activity categorization task.
In spite of using less
supervision at training time, our method outperforms state-of-the-art
methodologies that are trained with full spatial supervision.

\begin{table}[tb]
\footnotesize
\centering
\begin{tabular}{|l|c|}
\hline
\textbf{Algorithm} & \textbf{Accuracy}\\
\hline
Base model + GC, GEO desc. only, spatial supervision & 88.5\%\\
Base model + GC, with spatial supervision &  91.8\% \\
Our full model, no spatial supervision (latent $\vec{v}$) & 91.1\%\\
\hline
Lillo \etal \cite{Lillo2014} (without GC) & 85.7\% \\
Cao et al. \cite{cao2015spatio} & 79.0\% \\
\hline
\end{tabular}
\caption{
\footnotesize
Recognition accuracy in the Composable Activities
dataset.}
\label{tab:composable}
\end{table}

\subsection{Action Recognition in RGB Videos}
Our experiments so far have evaluated the performance of our model
in the task of human action recognition in RGBD videos.
In this experiment, we explore the use of our model in the problem of human
action recognition in RGB videos. For this purpose, we use the sub-JHMDB
dataset \cite{Jhuang2013}, which focuses on videos depicting 12 actions and
where most of the actor body is visible in the image frames.
In our validation, we use the 2D body pose configurations provided by the
authors and compare against previous methods that also use them. Given that
this dataset only includes 2D image coordinates for each body joint, we obtain
the geometric descriptor by adding a depth coordinate with a value $z = d$ to
joints corresponding to wrist and knees, $z = -d$ to elbows, and $z = 0$ to other joints,
so we can compute angles between segments, using $d = 30$ fixed with cross-validation. We summarize the results in Table
\ref{tab:subjhmdb},
which shows that our method outperforms alternative state-of-the-art techniques.

\begin{table}[tb]
\footnotesize
\centering
\begin{tabular}{|l|c|}
\hline
\textbf{Algorithm} & \textbf{Accuracy}\\
\hline
Our model &  77.5\% \\
\hline
Huang et al. \cite{Jhuang2013} & 75.6\% \\
Ch\'eron et al. \cite{Cheron2015} & 72.5\%\\
\hline
\end{tabular}
\caption{\footnotesize
Recognition accuracy in the sub-JHMDB dataset.}
\label{tab:subjhmdb}
\end{table}

\subsection{Spatio-temporal Annotation of Atomic Actions}
In this experiment, we study the ability of our model to provide spatial and
temporal annotations of relevant atomic actions. Table \ref{tab:annotation}
summarizes our results. We report precision-recall rates
for the spatio-temporal annotations predicted by our model in the
testing videos (first and second rows). Notice that this is a
very challenging task. The testing videos do no provide any label, and
the model needs to predict both, the temporal extent of each action and the
body regions associated with the execution of each action. Although the
difficulty of the task, our model shows satisfactory results being able to
infer suitable spatio-temporal annotations.

We also study the capability of the model to provide spatial and temporal
annotations during training. In our first experiment, each video
is provided
with the temporal extent of each action, so the model only needs to infer the
spatial annotations (third row in Table \ref{tab:annotation}). In a
second experiment, we do not provide any temporal or spatial annotation,
but only the global action label of each video (fourth row in Table
\ref{tab:annotation}). In both experiments, we observe that the model is
still able to infer suitable spatio-temporal annotations.

\begin{table}[tb]
\footnotesize
\centering
\begin{tabular}{|l|c|c|c|}
\hline
\textbf{Videos} & \textbf{Annotation inferred} & \textbf{Precision} & \textbf{Recall}\\
\hline
Testing set & Spatio-temporal, no GC   & 0.59 & 0.77 \\
Testing set & Spatio-temporal   & 0.62 & 0.78 \\
\hline
Training set & Spatial only & 0.86 & 0.90\\
Training set & Spatio-temporal & 0.67 & 0.85 \\
\hline
\end{tabular}
\caption{
\footnotesize
Atomic action annotation performances in the Composable Activities
dataset. The results show that our model is able to recover spatio-temporal
annotations both at training and testing time.}
\label{tab:annotation}
\end{table}

\subsection{Effect of Model Components}
In this experiment,
we study the contribution of key components of the
proposed model. First, using the sub-JHMDB dataset,
we measure the impact of three components of our model: garbage collector for
motion poselets (GC), multimodal modeling of actionlets, and use of latent
variables to infer spatial annotation about body regions (latent $\vec{v}$). Table
\ref{tab:components} summarizes our experimental results.
Table \ref{tab:components} shows that the full version
of our model achieves the best performance, with each of the components
mentioned above contributing to the overall success of the method.

\begin{table}[tb]
\footnotesize
\centering
\begin{tabular}{|l|c|}
\hline
\textbf{Algorithm} & \textbf{Accuracy}\\
\hline
Base model, GEO descriptor only & 66.9\%\\
Base Model & 70.6\%\\
Base Model + GC & 72.7\% \\
Base Model + Actionlets & 75.3\%\\
Our full model (Actionlets + GC + latent $\vec{v}$) &  77.5\% \\
\hline
\end{tabular}
\caption{
\footnotesize
Analysis of contribution to recognition performance from
each model component in the sub-JHMDB dataset.}
\label{tab:components}
\end{table}

Second, using the Composable Activities dataset, we also analyze the
contribution of the proposed self-paced learning scheme for initializing and
training our model. We summarize our results in
Table \ref{tab:initialization} by reporting action
recognition accuracy under different initialization schemes: i) Random: random
initialization of latent variables $\vec{v}$, ii) Clustering: initialize
$\vec{v}$ by first computing a BoW descriptor for the atomic action intervals
and then perform $k$-means clustering, assigning the action intervals to the
closer cluster center, and iii) Ours: initialize $\vec{v}$ using the proposed
self-paced learning scheme. Our proposed initialization scheme helps the model to achieve its best
performance.

\begin{table}[tb]
\footnotesize
\centering
\begin{tabular}{|l|c|}
\hline
\textbf{Initialization Algorithm} & \textbf{Accuracy}\\
\hline
Random   & 46.3\% \\
Clustering   & 54.8\% \\
Ours   & 91.1\% \\
\hline
Ours, fully supervised & 91.8\%\\
\hline
\end{tabular}
\caption{
\footnotesize
Results in Composable Activities dataset, with latent $\vec{v}$ and different initializations. }
\label{tab:initialization}
\end{table}

\subsection{Qualitative Results}
Finally, we provide a qualitative analysis of
relevant properties of our model. Figure \ref{fig:poselets_img}
shows examples of moving poselets learned in the Composable
Activities dataset. We observe that each moving poselet captures
a salient body configuration that helps to discriminate among atomic
actions. To further illustrate this, Figure \ref{fig:poselets_img}
indicates the most likely underlying atomic action for each moving poselet.
Figure \ref{fig:poselets_skel} presents a similar analysis for moving
poselets learned in the MSR-Action3D dataset.

We also visualize the action annotations produced by our model.
Figure \ref{fig:actionlabels} (top) shows the action labels associated
with each body part in a video from the Composable Activities dataset.
Figure \ref{fig:actionlabels} (bottom) illustrates per-body part action
annotations for a video in the Concurrent Actions dataset. These
examples illustrate the capabilities of our model to correctly
annotate the body parts that are involved in the execution of each action,
in spite of not having that information during training.

\begin{figure}[tb]
\begin{center}
\scriptsize
 Motion poselet \#4 - most likely action: talking on cellphone\\
 \includegraphics[trim=0 0 0 0.35cm, clip, width=0.49\textwidth]{Fig/poselets1}

 Motion poselet \#7 - most likely action: erasing on board\\
 \includegraphics[trim=0 0 0 0.35cm, clip, width=0.49\textwidth]{Fig/poselets2}

 Motion poselet \#19 - most likely action: waving hand\\
 \includegraphics[trim=0 0 0 0.35cm, clip, width=0.49\textwidth]{Fig/poselets3}
\end{center}
\caption{
\footnotesize
Moving poselets learned from the Composable Activities
dataset.}
\label{fig:poselets_img}
\end{figure}

\begin{figure}[tb]
\begin{center}
\scriptsize
 Motion poselet \#16 - most likely action: tennis swing\\
 \includegraphics[trim=0 0 0cm 0cm, clip, width=0.49\textwidth]{Fig/poselets4}

 Motion poselet \#34 - most likely action: golf swing\\
 \includegraphics[trim=0 0 0cm 0cm,clip, width=0.49\textwidth]{Fig/poselets5}

 Motion poselet \#160 - most likely action: bend\\
 \includegraphics[trim=0 0 0cm 0cm, clip, width=0.49\textwidth]{Fig/poselets6}

\end{center}
\caption{
\footnotesize
Moving poselets learned from the MSR-Action3D
dataset.}
\label{fig:poselets_skel}
\end{figure}

\begin{figure}[tb]
\begin{center}
\scriptsize
\includegraphics[]{Fig/labels_acciones}
\end{center}
\caption{
\footnotesize
Automatic spatio-temporal annotation of atomic actions. Our method
detects the temporal span and spatial body regions that are involved in
the performance of atomic actions in videos.}
\label{fig:actionlabels}
\end{figure}

\begin{comment}

[GENERAL IDEA]

What we want to show:
\begin{itemize}
\item Show tables of results that can be useful to compare the model.
\item Show how the model is useful for videos of simple and composed actions, since now the level of annotations is similar.
\item Show how the inference produces annotated data (poses, actions, etc). In particular, show in Composable Activities and Concurrent actions how the action compositions are handled by the model without post-processing.
\item Show results in sub-JHMDB,showing how the model detects the action in the videos and also which part of the body performs the action (search for well-behaved videos). It could be interesting to show the annotated data over real RGB videos.
\item Show examples of poses (like poselets) and sequences of 3 or 5 poses for actions (Actionlets?)
\end{itemize}

\subsection{Figures}
The list of figures should include:
\begin{itemize}
\item A figure showing the recognition and mid-level labels of Composable Activities, using RGB videos
\item Comparison of action annotations, real v/s inferred in training set, showing we can recover (almost) the original annotations.
\item Show a figure similar to Concurrent Actions paper, with a timeline showing the actions in color. We can show that our inference is more stable than proposed in that paper, and it is visually more similar to the ground truth than the other methods.
\item Show a figure for sub-JHMDB dataset, where we can detect temporally and spatially the action without annotations in the training set.
\item Show Composable Activities and sub-JHMDB  the most representative poses and actions.
\end{itemize}

\paragraph{Composable Activities Dataset}
In this dataset we show several results.
(1) Comparing TRAJ descriptor (HOF over trajectory);
(2) Compare the results using latent variables for action assignations to
regions, with different initializations;
(3) Show results of the annotations of the videos in inference.

We must include figures comparing the real annotations
and the inferred annotations for training data, to show we are able to get the
annotations only from data.

\subsection{Recognition of composable activities}
\label{subsec:experiments_summary}

\subsection{Impact of including motion features}
\label{subsec:exp_motionfeats}

\subsection{Impact of latent spatial assignment of actions}
\label{subsec:exp_vlatent}

\subsection{Impact of using multiple classifiers per semantic action}
\label{subsec:exp_multiple}

\subsection{Impact of handling non-informative poses}
\label{subsec:exp_non_info_handling}
\end{comment}

\begin{comment}
\subsection{CAD120 Dataset}
The CAD120 dataset is introduced in \cite{Koppula2012}. It is composed of 124
videos that contain activities in 10 clases performed by 4 actors. Activities
are related to daily living: \emph{making cereal}, \emph{stacking objects}, or
\emph{taking a meal}. Each activity is composed of simpler actions like
\emph{reaching}, \emph{moving}, or \emph{eating}. In this database, human-object
interactions are an important cue to identify the actions, so object
locations and object affordances are provided as annotations. Performance
evaluation is made through leave-one-subject-out cross-validation. Given
that our method does not consider objects, we use only
the data corresponding to 3D joints of the skeletons. As shown in Table
\ref{Table-CAD120},
our method outperforms the results reported in
\cite{Koppula2012} using the same experimental setup. It is clear that using
only 3D joints is not enough to characterize each action or activity in this
dataset. As part of our future work, we expect that adding information related
to objects will further improve accuracy.
\begin{table}
\centering
{\small
\begin{tabular}{|c|c|c|}
\hline
\textbf{Algorithm} & \textbf{Average precision} & \textbf{Average recall}\\
\hline
Our method &  32.6\% & 34.58\% \\
\hline
\cite{Koppula2012} &   27.4\% & 31.2\%\\
\cite{Sung2012} &  23.7\%  &  23.7\% \\
\hline
\end{tabular}
}
\caption{Recognition accuracy of our method compared to state-of-the-art methods
using CAD120 dataset.}
\label{Table-CAD120}
\end{table}
\end{comment}

\subsection{Latent spatial actions for hierarchical action detection}

\subsection{Hierarchical activity model}

Suppose we have a video $D$ with $T$ frames, each frame described by a feature vector $x_t$. Assume we have available $K$ classifiers$\{w_k\}_{k=1}^K$ over the frame descriptors, such that each frame descriptor can be associated to a single classifier. If we choose the maximum response for every frame, encoded as $z_t = \argmax_k\{w_k^\top x_t\}$,  we can build a BoW representation to feed linear action classifiers $\beta$, computing the histogram $h(Z)$ of $Z = \{z_1,z_2,\dots,z_T\}$ and using these histograms as a feature vector for the complete video to recognize single actions. Imagine now that we would like to use the scores of the maximum responses, $w_{z_t}^\top x_t$ as a potential to help discriminating between videos that present reliable poses from videos that does not. We can build a joint energy function, combining the action classifier score and the aggregated frame classifier scores, as
\begin{equation}
\label{eq:2-levels}
\begin{split}
E(D) &= \beta_{a}^\top h(Z) + \sum_{t=1}^T w_{z_t}^\top x_t \\ & = \sum_{t=1}^T\sum_{k=1}^K\left(\beta_{a,k} + w_k^\top x_t \right)\delta(z_t=k)
\end{split}
\end{equation}
What is interesting of Eq. (\ref{eq:2-levels}) is that it every term in the sum is tied for the value of $z_t$, creating a model such that all its components depends of the labeling $Z$. We can expand the previous model to more levels using the same philosophy. In fact, for a new level, we could create a new indicator $v_t$ for every frame that indicates the election of which classifier $\beta$ will be used (the same as $z_t$ indicates which classifier of $w$). If we name $w$ as \emph{pose classifiers}, and $\beta$ as \emph{action classifiers}, we can create a hierarchical model where multiple poses and actions can be present in a single video. Supposing we have $A$ actions; the energy for a three-level hierarchy could be, for an \emph{activity} $l$,
\begin{equation}
E(D) =\alpha_l^\top h(V) + \sum_{a=1}^A \beta_{a}^\top h^a(Z,V) + \sum_{t=1}^T w_{z_t}^\top x_t
\end{equation}
where $h^a(Z,V)$ refers to the BoW representation of $Z$ for those frames labeled as action $v_t = a$.

[NEW MODEL]

Recent work in action recognition \cite{Cheron2015,Tao2015, Wang2011,Jhuang2013} shows a resurgence of describing human actions as a collection of dynamic spatial parts that resembles Poselets. In line with these research, we split the human body into $R$ semantic regions. As modeling actions using the whole body is hard, separating the body into groups of limbs helps in recognition of actions, specially for complex datasets \cite{Tao2015}. Our wiew is that while poses are in general well defined in most research, little effort has been made to mine actions from videos, in terms of detecting the temporal spanning (action detection) and action localization. In addition to the fact that most action datasets are only single actions, there is a lack of research in the general setup where actions are combined in the same video. Nevertheless, a few works have noticed that humans usually performs complex action in real life \cite{Wei2013, Lillo2014}, providing their own datasets based in RGB-D cameras. In our work, we aim to group both worlds of single and composed actions in a single hierarchical model of three semantic levels, and using human body regions to improve the representativeness.

During training, we assume there is temporal annotations of actions. As we want our model to perform action localization, we model the action assignments $V_r$ in each region as latent variables during training, allowing the model to infer which human part execute the action without needing this kind of annotations in the training set, including a model for the initialization of action labels. In this way, we advance from a simple detection problem to infer also \emph{how} the subject executes the action, important in surveillance applications, health monitoring, between others. We also expand the modeling of recurrent patterns of poses to construct a general model for shared actions, aiming to handle multimodal information, which is produced by actions with the same label but with different execution patterns, or by changes in representation of actions such as varying camera view. We handle this problem by augmenting the number of action classifiers, where each original action acts as a parent node of several non-overlapping child actions. Finally, as we are using local information for poses, some frames could be noisy or representing an uncommon pose, not useful to build the pose models. We attack this issue by adding a garbage collector for poses, where only the most-informative poses are used by pose classifiers during learning. We describe these contributions in the following paragraphs.

\paragraph{[EDIT] Latent assignments of actions to human regions}

Knowing the parts of the body involved in the actions is highly appealing. Suppose we have $M$ videos, each video annotated with $Q_m$ action intervals. Each action interval can be associated with any number of regions, from $1$ to all $R$ regions. For example,  a \emph{waving hand} action could be associated only with \emph{right\_arm}, while the action \emph{jogging} could be associated with the whole body. We want to learn the associations of actions and human parts for training videos, and we build these associations using latent variables. The main problem to solve is to how to get a proper initialization for actions, since there is a very high chance to get sticked in a local minimum far away of the optimum, producing bad results.

Our first contribution is a method to get a proper initialization of fine-grained spatial action labels, knowing only the time span of the actions. Using the known action intervals,  we formulate the problem of action to region assignment as an optimization problem, constrained using structural information: the actions intervals must not overlap in the same region, and all the action intervals must be present at least in one region. We formulate this labeling problem as a binary Integer Linear Programming (ILP) problem. We define as $v_{r,q}^m=1$ when the action interval $q \in \{1,\dots,Q_m\}$ appears in region $r$ in the video $m$, and $v_{r,q}^m=0$ otherwise. We assume we have pose labels $z_{t,r}$ in each frame, independent for each region, learned via clustering the poses for all frames in all videos. For an action interval $q$, we use as descriptor the histogram of pose labels for each region in the action interval, defined for the video $m$ as $h_{r,q}^m$ . We can solve the problem of finding the correspondence between action intervals and regions in a formulation similar to $k$-means, using the structure of the problem as constraints in the labels, and using $\chi^2$ distance between the action interval descriptors and the cluster centers:
\begin{equation}
\begin{split}
P1) \quad \min_{v,\mu} &\sum_{m=1}^M  \sum_{r=1}^R \sum_{q=1}^{Q_m}  v_{r,q}^m d( h_{r,q}^m - \mu_{a_q}^r) -\frac{1}{\lambda} v_{r,q}^m\\
 \text{s. to}
\quad
& \sum_{r=1}^R v_{r,q}^m \ge 1\text{, }\forall q\text{, }\forall m \\
& v_{r,q_1}^m + v_{r,q_2}^m \le 1 \text{ if } q_1\cap q_2 \neq \emptyset \text{, }\forall r\text{, }\forall m\\
& v_{r,q}^m \in \{0,1\}\text{, }\forall q\text{, }\forall{r}\text{, }\forall m
\end{split}
\end{equation}
with
\begin{equation}
d( h_{r,q}^m - \mu_{a_q}^r) = \sum_{k=1}^K (h_{r,q}^m[k] - \mu_{a_q}^r[k])^2/(h_{r,q}^m[k] +\mu_{a_q}^r[k]).
\end{equation}

$\mu_{a_q}^r$ are computed as the mean of the descriptors with the same action label within the same region. We solve $P1$ iteratively as $k$-means,  finding the cluster centers for each region $r$, $\mu_{a}^r$ using the labels $v_{r,q}^m$, and then finding the best labeling given the cluster centers, solving an ILP problem. Note that the first term of the objective function is similar to a $k$-means model, while the second term resembles the objective function of \emph{self-paced} learning as in \cite{Kumar2010}, fostering to balance between assigning a single region to every action, towards assigning all possible regions to the action intervals when possible.

[IL: INCLUDE  FIGURE TO SHOW P1 GRAPHICALLY]

We describe the further changes in the hierarchical model of \cite{Lillo2014} in the learning and inference sections.
  \paragraph{[EDIT] Representing semantic actions with multiple atomic sequences}.

As the poses and atomic actions in \cite{Lillo2014} model are shared, a single classifier is generally not enough to model multimodal representations, that occur usually in  complex videos. We modify the original hierarchical model of \cite{Lillo2014} to include multiple linear classifiers per action. We create two new concepts: \textbf{semantic actions}, that refer to actions \emph{names} that compose an activity; and \textbf{atomic sequences}, that refers to the sequence of poses that conform an action. Several atomic sequences can be associated to a single semantic action, creating disjoint sets of atomic sequences, each set associated to a single semantic action.  The main idea is that the action annotations in the datasets are associated to semantic actions, whereas for each semantic action we learn several atomic sequence classifiers. With this formulation, we can handle the multimodal nature of semantic actions, covering the changes in motion, poses , or even changes in meaning of the action according to the context (e.g. the semantic action ``open'' can be associated to opening a can, opening a door, etc.).

Inspired by \cite{Raptis2012}, we first use the \emph{Cattell's Scree test} for finding a suitable number of atomic sequence for every semantic action. Using the semantic action labels, we compute a descriptor for every interval using normalized histograms of pose labels. Then, for a particular semantic action $u$, we compute the the eigenvalues $\lambda_u$ of the affinity matrix of the semantic action descriptors, using $\chi^2$ distance. For each semantic action $u \in \{1,\dots,U\}$ we find the number of atomic sequences $G_u$ as $G_u = \argmin_i \lambda_{i+1}^2 / (\sum_{j=1}^i \lambda_j) + c\cdot i$, with $c=2\cdot 10^{-3}$. Finally, we cluster the descriptors corresponding to each semantic action using k-means, using a different number of clusters for each semantic action $u$ according to $G_u$. This approach generates non-overlapping atomic sequences, each associated to a single semantic action.

To transfer the new labels to the model, we define $u(v)$ as the function that given the atomic sequence label $v$, returns the corresponding semantic action label $u$. The energy for the activity level is then
\begin{equation}
E_{\text{activity}} =  \sum_{u=1}^U\sum_{t=1}^T \alpha_{y,u}\delta(u(v_t)=u)
\end{equation}

For the action and pose labels the model remains unchanged. Using the new atomic sequences allows a richer representation for actions, while in he activity level, several atomic sequences will map to a single semantic action. This behavior resembles a max-pooling operation, where we will choose at inference the atomic sequences that best describe the performed actions in the video, keeping the semantics of the original labels.

\paragraph{Towards a better representation of poses: adding a garbage collector}

The model in \cite{Lillo2014} uses all poses to feed action classifiers. Out intuition is that only a subset of poses in each video are really discriminative or informative for the actions performed, while there is plenty of poses that corresponds to noisy or non-informative ones. [EXPAND] Our intuition is that low-scored frames in terms of poses (i.e. a low value of $w_{z_t}^\top x_t$ in Eq. (\ref{eq:energy2014})) make the same contribution as high-scored poses in higher levels of the model, while degrading the pose classifiers at the same time since low-scored poses are likely to be related to non-informative frames. We propose to include a new pose, to explicitly handling those low-scored frames, keeping them apart for the pose classifiers $w$, but still adding a fixed score to the energy function to avoid normalization issues and to help in the specialization of pose classifiers. We call this change in the model a \emph{garbage collector} since it handles all low-scores frames and group them having a fixed energy score $\theta$. In practice, we use a special pose entry $K+1$ to identify the non-informative poses. The equation representing the energy for pose level is
\begin{equation} \label{Eq_poseEnergy}
E_{\text{poses}}  = \sum_{t=1}^T \left[  {w_{z_t}}^\top x_{t}\delta(z_{t} \le  K) + \theta
\delta(z_{t}=K+1)\right]
\end{equation}
where $\delta(\ell) = 1$ if $\ell$ is true and $\delta(\ell) = 0$ if
$\ell$ is false. The action level also change its energy:
\begin{equation}
\begin{split}
 \label{Eq_actionEnergy}
E_{\text{actions}} =  \sum_{t=1}^T \sum_{a=1}^A \sum_{k=1}^{K+1}  \beta_{a,k} \delta(z_t = k) \delta(v_t = a).
\end{split}
\end{equation}

\begin{comment}
Integrating all contribution detailed in previous sections, the model is written as:
Energy function:
\begin{equation}
E = E_{\text{activity}} + E_{\text{action}} + E_{\text{pose}}
  + E_{\text{action transition}} + E_{\text{pose transition}}.
\end{equation}

\begin{equation}
E_{\text{poses}}  = \sum_{t=1}^T \left[  {w_{z_t}}^\top x_{t}\delta(z_{t} \le  K) + \theta
\delta(z_{t}=K+1)\right]
\end{equation}

\begin{equation}
E_{\text{actions}} =  \sum_{t=1}^T \sum_{a=1}^A \sum_{k=1}^{K+1}  \beta_{a,k} \delta(z_t = k) \delta(v_t = a).
\end{equation}

\begin{equation}
h_g^{r}(U) = \sum_{t} \delta_{u_{t,r}}^g
\end{equation}

So the energy in the activity level is
\begin{equation}
E_{\text{activity}} = \sum_{r} {\alpha^r_{y}}^\top h^{r}(U) = \sum_{r,g,t}  \alpha^r_{y,g} \delta_{u_{t,r}}^g
\end{equation}

\begin{equation}
E_{\text{action transition}} = \sum_{r,a,a'}  \gamma^r_{a',a} \sum_{t} \delta_{v_{t-1,r}}^{a'}\delta_{v_{t,r}}^a
\end{equation}

\begin{equation}
E_{\text{pose transition}} =\sum_{r,k,k'}  \eta^r_{k',k}\sum_{t}\delta_{z_{t-1,r}}^{k'}\delta_{z_{t,r}}^{k}
\end{equation}
\end{comment}

\subsection{Inference}
\label{subsec:inference}
The input to the inference algorithm is a new video sequence with features
$\vec{x}$. The task is to infer the best complex action label $\hat y$, and to
produce the best labeling of actionlets $\hat{\vec{v}}$ and motion poselets $\hat{\vec{z}}$.
{\small
\begin{equation}
  \hat y, \hat{\vec{v}}, \hat{\vec{z}} = \argmax_{y, \vec{v},\vec{z}} E(\vec{x}, \vec{v}, \vec{z}, y)
\end{equation}}
We can solve this by exhaustively enumerating all values of complex actions $y$, and solving for $\hat{\vec{v}}$ and $\hat{\vec{z}}$ using:
\small
\begin{equation}
\begin{split}
 \hat{\vec{v}}, \hat{\vec{z}} | y ~ =~ &   \argmax_{\vec{v},\vec{z}} ~   \sum_{r=1}^R \sum_{t=1}^T \left( \alpha^r_{y,u(v{(t,r)})}
                  + \beta^r_{v_{(t,r)},z_{(t,r)}}\right. \\
				&\quad\quad \left.+ {w^r_{z_{(t,r)}}}^\top x_{t,r} \delta(z_{(t,r)} \le K)  + \theta^r \delta_{z_{(t,r)}}^{K+1} \right. \\
				& \quad\quad \left.+ \gamma^r_{v_{({t-1},r)},v_{(t,r)}} + \eta^r_{z_{({t-1},r)},z_{(t,r)}}  \vphantom{{w^r_{z_{(t,r)}}}^\top x_{t,r}} \right). \\
\end{split}
\label{eq:classify_inference}
\end{equation}
\normalsize

\subsection{Learning} \label{subsec:learning}
\textbf{Initial actionlet labels.} An important step in the training process is
the initialization of latent variables. This is a challenging due to the lack
of spatial supervision: at each time instance, the available atomic actions can be associated with
any of the $R$ body regions.
We adopt the machinery of
self-paced
learning \cite{Kumar:EtAl:2010} to provide a suitable solution and
formulate the association between actions and body regions as an
optimization problem. We constrain this optimization using two structural
restrictions:
i) atomic actions intervals must not overlap in the same region, and
ii) a labeled atomic action must be present at least in one region. We
formulate the labeling
process as a binary Integer Linear Programming (ILP) problem, where we define
$b_{r,q}^m=1$ when action interval $q \in \{1,\dots,Q_m\}$ is active in region
$r$ of video $m$; and $b_{r,q}^m=0$ otherwise. Each action interval $q$ is
associated with a single atomic action. We assume that we have initial
motion poselet labels
$z_{t,r}$ in each frame and region.
We describe the action interval $q$ and region $r$ using
the histogram $h_{r,q}^m$ of motion poselet labels. We can find
the correspondence between action intervals and regions using a formulation
that resembles the operation of$k$-means, but using the
structure of the problem to constraint the labels:
\small
\begin{equation}
\begin{split}
\text{P1}) \quad \min_{b,\mu} &\sum_{m=1}^M  \sum_{r=1}^R \sum_{q=1}^{Q_m}  b_{r,q}^m
d( h_{r,q}^m - \mu_{a_q}^r) -\frac{1}{\lambda} b_{r,q}^m\\
 \text{s.t.}
\quad
& \sum_{r=1}^R b_{r,q}^m \ge 1\text{, }\forall q\text{, }\forall m \\
& b_{r,q_1}^m + b_{r,q_2}^m \le 1 \text{ if } q_1\cap q_2 \neq \emptyset
\text{,
}\forall r\text{, }\forall m\\
& b_{r,q}^m \in \{0,1\}\text{, }\forall q\text{, }\forall{r}\text{, }\forall m
\end{split}
\end{equation}
with
\begin{equation}
d( h_{r,q}^m - \mu_{a_q}^r) = \sum_{k=1}^K (h_{r,q}^m[k] -
\mu_{a_q}^r[k])^2/(h_{r,q}^m[k] +\mu_{a_q}^r[k]).
\end{equation}
\normalsize
Here, $\mu_{a_q}^r$ are the means of the descriptors with action
label $a_q$ within region $r$. We solve $\text{P1}$ iteratively using a block coordinate
descending scheme, alternating between solving $b_{r,q}^m$ with $\mu_{a}^r$
fixed, which has a trivial solution; and then fixing $\mu_{a}^r$ to solve
$b_{r,q}^m$, relaxing $\text{P1}$ to solve a linear program. Note that the second term
of the objective function in $\text{P1}$ resembles the objective function of
\emph{self-paced} learning \cite{Kumar:EtAl:2010}, managing the balance between
assigning a single region to every action or assigning all possible regions to
the respective action interval.

\textbf{Learning model parameters.}
We formulate learning the model parameters as a Latent Structural SVM
problem \cite{Yu:Joachims:2010}, with latent variables for motion
poselets $\vec{z}$ and actionlets $\vec{v}$. We find values for parameters in
equations
(\ref{eq:motionposelets}-\ref{eq:actionletstransition}),
slack variables $\xi_i$, motion poselet labels $\vec{z}_i$, and actionlet labels $\vec{v}_i$,
by solving:
{\small
\begin{equation}
\label{eq:big_problem}
\min_{W,\xi_i,~i=\{1,\dots,M\}}    \frac{1}{2}||W||_2^2 + \frac{C}{M} \sum_{i=1}^M\xi_i ,
\end{equation}}
where
{\small \begin{equation}
W^\top=[\alpha^\top, \beta^\top, w^\top, \gamma^\top, \eta^\top, \theta^\top],
\end{equation}}
and
{\small
\begin{equation} \label{eq:slags}
\begin{split}
\xi_i = \max_{\vec{z},\vec{v},y}  \{  & E(\vec{x}_i, \vec{z}, \vec{v}, y) + \Delta( (y_i,\vec{v}_i), (y, \vec{v})) \\
 & - \max_{\vec{z}_i}{ E(\vec{x}_i, \vec{z}_i, \vec{v}_i, y_i)} \}, \; \;\; i\in[1,...M].
\end{split}
\end{equation}}
In Equation (\ref{eq:slags}), each slack variable
$\xi_i$ quantifies the error of the inferred labeling for
video $i$. We solve Equation (\ref{eq:big_problem}) iteratively using the CCCP
algorithm \cite{Yuille:Rangarajan:03}, by solving for
latent labels $\vec{z}_i$ and $\vec{v}_i$ given model parameters $W$,
temporal atomic action annotations (when available), and labels of complex actions occurring in
training videos (see Section \ref{subsec:inference}). Then, we solve for
$W$ via 1-slack formulation using Cutting Plane algorithm
\cite{Joachims2009}.

The role of the loss function $\Delta((y_i,\vec{v}_i),(y,\vec{v}))$ is to penalize inference errors during
training. If the true actionlet labels are known in advance, the loss function is the same as in \cite{Lillo2014} using the actionlets instead of atomic actions:
\small \begin{equation}
\Delta((y_i,\vec{v}_i),(y,\vec{v})) = \lambda_y(y_i \ne y) + \lambda_v\frac{1}{T}\sum_{t=1}^T
\delta({v_t}_{i} \neq v_t),
\end{equation}
\normalsize
\noindent where ${v_t}_{i}$ is the true actionlet label. If the spatial ordering of actionlets is unknown (hence the latent
actionlet formulation), but the temporal composition is known, we can compute a
list $A_t$ of possible actionlets for frame $t$, and include that information
on the loss function as
\small \begin{equation}
\Delta((y_i,\vec{v}_i),(y,\vec{v})) = \lambda_y(y_i \ne y) + \lambda_v\frac{1}{T}\sum_{t=1}^T
\delta(v_t \notin A_t)
\end{equation}
\normalsize

\subsection{Body regions}
We divide the body pose into $R$ fixed spatial regions and independently compute
a pose feature vector for each region. Figure \ref{fig:skeleton_limbs_regions}
illustrates the case when $R = 4$ that we use in all our experiments. Our body
pose feature vector consists of the concatenation of two descriptors. At frame
$t$ and region $r$, a descriptor $x^{g}_{t,r}$ encodes geometric information
about the spatial configuration of body joints, and a descriptor $x^{m}_{t,r}$
encodes local motion information around each body joint position.
We use the geometric descriptor from \cite{Lillo2014}:
we construct six segments that connect pairs of joints at each
region\footnote{Arm segments: wrist-elbow, elbow-shoulder, shoulder-neck, wrist-shoulder, wrist-head, and neck-torso;  Leg segments: ankle-knee, knee-hip, hip-hip center, ankle-hip, ankle-torso and hip center-torso}
and compute 15 angles between those segments.
Also, three angles are calculated between a plane formed by three
segments\footnote{Arm plane: shoulder-elbow-wrist;  Leg plane: hip-knee-ankle} and
the remaining three non-coplanar segments, totalizing an 18-D geometric descriptor (GEO) for every region.
Our motion descriptor is based on tracking motion trajectories of key points
\cite{WangCVPR2011}, which in our case coincide with body joint positions.
We extract a HOF descriptor
using 32x32 RGB patches centered at the joint location for a temporal window of 15
frames. At each joint location, this produces a 108-D descriptor,
which we concatenate across all joints in each a region to obtain our motion descriptor. Finally,
we apply PCA to reduce the dimensionality of our concatenated motion descriptor
to 20. The final descriptor is the concatenation of the geometric and
motion descriptors, $x_{t,r} = [x_{t,r}^g ; x_{t,r}^m]$.

\subsection{Hierarchical compositional model}

We propose a hierarchical compositional model that spans three semantic
levels. Figure \ref{fig:overview} shows a schematic of our model. At the
top level, our model assumes that each input video has a single complex action
label $y$. Each complex action is composed of a
temporal and spatial arrangement of atomic actions with labels $\vec{u}=[u_1,\dots,u_T]$, $u_i \in \{1,\dots,S\}$.
In turn, each atomic action consists of several non-shared \emph{actionlets}, which correspond to representative sets of pose configurations for action identification, modeling the multimodality of each atomic action.
We capture actionlet assignments in $\vec{v}=[v_1,\dots,v_T]$, $v_i \in \{1,\dots,A\}$.
Each actionlet index $v_i$ corresponds to a unique and known actomic action label $u_i$, so they are related by a mapping $\vec{u} = \vec{u}(\vec{v})$. At the
intermediate level, our model assumes that each actionlet is composed of a
temporal arrangement of a subset from $K$ body poses, encoded in $\vec{z} = [z_1,\dots,z_T]$, $z_i \in \{1,\dots,K\}$,
where $K$ is a hyperparameter of the model.
These subsets capture pose geometry and local motion, so we call them \emph{motion poselets}.
Finally, at the bottom level, our model identifies motion poselets
using a bank of linear classifiers that are applied to the incoming frame
descriptors.

We build each layer of our hierarchical model on top of BoW
representations of labels. To this end, at the bottom level of our hierarchy, and for
each body region, we learn a dictionary of motion poselets. Similarly, at the mid-level of our hierarchy, we learn a dictionary of actionlets, using the BoW representation of motion poselets as inputs. At each of these levels,
spatio-temporal activations of the respective dictionary words are used
to obtain the corresponding histogram encoding the BoW representation.
The next two sections provide
details on the process to represent and learn the dictionaries of motion
poselets and actionlets. Here we discuss our
integrated hierarchical model.

We formulate our hierarchical model using an energy function.
Given a video of $T$ frames corresponding to complex action $y$ encoded by descriptors $\vec{x}$, with the label vectors $\vec{z}$ for motion poselets,
$\vec{v}$ for actionlets and $\vec{u}$ for atomic actions, we
define an energy function for a video as:
\small
\begin{align}\label{Eq_energy}
E(\vec{x},&\vec{v},\vec{z},y) =  E_{\text{motion poselets}}(\vec{z},\vec{x}) \nonumber \\&+ E_{\text{motion poselets BoW}}(\vec{v},\vec{z}) +
E_{\text{atomic actions BoW}}(\vec{u}(\vec{v}),y) \nonumber \\
& + E_{\text{motion poselets transition}}(\vec{z}) + E_{\text{actionlets
transition}}(\vec{v}).
\end{align}
\normalsize
Besides the BoW representations and motion poselet classifiers
described above, Equation (\ref{Eq_energy}) includes
two energy potentials that encode information related to
temporal
transitions between pairs of motion poselets ($E_{\text{motion poselets
transition}}$) and
actionlets ($E_{\text{actionlets transition}}$).
The energy potentials are given by:
{\small
\begin{align}
\label{eq:motionposelets}
&E_{\text{mot. poselet}}(\vec{z},\vec{x})  = \sum_{r,t}  \left[ \sum_{k} {w^r_k}^\top
x_{t,r}\delta_{z_{(t,r)}}^{k} + \theta^r \delta_{z_{(t,r)}}^{K+1}\right] \\
&E_{\text{mot. poselet BoW}}(\vec{v},\vec{z})  = \sum_{r,a,k} {\beta^r_{a,k}}\delta_{v_{(t,r)}}^{a}\delta_{z_{(t,r)}}^{k}\\
\label{eq:actionlets_BoW}
&E_{\text{atomic act. BoW}}(\vec{u}(\vec{v}),y) =\sum_{r,s} {\alpha^r_{y,s}}\delta_{u(v_{(t,r)})}^{s} \\
&E_{\text{mot. pos. trans.}}(\vec{z})  =
\sum_{r,k_{+1},k'_{+1}} \eta^r_{k,k'}
\sum_{t} \delta_{z_{(t-1,r)}}^{k}\delta_{z_{(t,r)}}^{k'} \\
\label{eq:actionletstransition}
&E_{\text{acttionlet trans.}}(\vec{v})  =\sum_{r,a,a'} \gamma^r_{a,a'}
\sum_{t}
\delta_{v_{(t-1,r)}}^{a}\delta_{v_{(t,r)}}^{a'}
\end{align}
}

Our goal is to
maximize $E(\vec{x},\vec{v},\vec{z},y)$, and obtain the
spatial and temporal arrangement
of motion poselets $\vec{z}$ and actionlets $\vec{v}$, as well as, the underlying
complex action $y$.

In the previous equations, we use $\delta_a^b$ to indicate the Kronecker delta function $\delta(a = b)$, and use indexes $k \in \{1,\dots,K\}$ for motion poselets, $a \in \{1,\dots,A\}$ for actionlets, and $s \in \{1,\dots,S\}$ for atomic actions.
In the energy term for motion poselets,
$w^r_k$ are a set of $K$ linear pose classifiers applied to frame
descriptors $x_{t,r}$, according to the label of the latent variable $z_{t,r}$.
Note that there is a special label $K+1$; the role of this label will be
explained in Section \ref{subsec:garbage_collector}.
In the energy potential associated to
the BoW representation for motion poselets, $\vec{\beta}^r$ denotes a set of $A$
mid-level classifiers, whose inputs are histograms of motion
poselet labels at those frame annotated as actionlet $a$. At the highest level,
$\alpha^r_{y}$ is a linear classifier associated with complex action $y$, whose
input is the histogram of atomic action labels,
which are related to actionlet assignments by the mapping function $\vec{u}(\vec{v})$. Note that all classifiers
and labels here correspond to a single region $r$. We add the contributions of all
regions to compute the global energy of the video. The transition terms act as
linear classifiers $\eta^r$ and $\gamma^r$ over histograms of temporal transitions of motion poselets
and temporal transitions of actionlets respectively. As we have a special label $K+1$ for motion poselets, the summation index
$k_{+1}$ indicates the interval $\lbrack 1,\dots,K+1 \rbrack$.

\subsection{Learning motion poselets}
In our model, motion poselets are learned by treating them as latent variables
during training. Before training, we fix the number of motion poselets per region to $K$.
In every region $r$, we learn an independent
set of pose classifiers $\{w^r_k\}_{k=1}^K$, initializing the motion poselet
labels using the $k$-means algorithm. We learn pose classifiers,
actionlets and complex actions classifiers jointly, allowing the model to discover
discriminative motion poselets useful to detect and recognize complex actions.
As shown in previous work, jointly learning linear
classifiers to identify body parts and atomic actions improves recognition
rates \cite{Lillo2014,Wang2008}, so here we follow a similar hierarchical
approach, and integrate learning
of motion poselets with the learning of actionlets.

\subsection{Learning actionlets}
\label{sec:learningactionlets}
A single linear classifier does not offer enough flexibility to identify atomic
actions that exhibit high visual variability. As an example, the atomic action
``open'' can be associated with ``opening a can'' or ``opening a
book'', displaying high variability in action execution. Consequently, we
augment our hierarchical model including multiple classifiers to
identify different modes of action execution.

Inspired by \cite{Raptis2012}, we use the \emph{Cattell's Scree test} to
find a suitable number of actionlets to model each atomic
action. Specifically, using the atomic action labels, we compute a descriptor
for every video interval using
normalized histograms of initial pose labels obtained with $k$-means. Then, for a particular atomic action
$s$, we compute the eigenvalues $\lambda(s)$ of the affinity matrix of the
atomic action descriptors, which is build using $\chi^2$ distance. For each
atomic action
$s \in \{1,\dots,S\}$, we find the number of actionlets $G_s$ as $G_s =
\argmin_i {\lambda(s)}_{i+1}^2 / (\sum_{j=1}^i {\lambda(s)}_j) + c\cdot i$, with $c=2\cdot
10^{-3}$. Finally, we cluster the descriptors from each atomic
action $s$ running $k$-means with $k = G_s$. This scheme generates
a set of non-overlapping actionlets to model each single atomic
action. In our experiments, we notice that the number of actionlets used to
model each atomic action varies typically from 1 to 8.

To transfer the new labels to the model, we define $u(v)$ as a function that
maps from actionlet label $v$ to the corresponding atomic action label
$u$. A dictionary of actionlets provides a richer representation for actions,
where several actionlets will map to a single atomic action. This behavior
resembles a max-pooling operation, where at inference time we will choose the
set of actionlets that best describe the performed actions in the video, keeping
the semantics of the original atomic action labels.

\subsection{A garbage collector for motion poselets}
\label{subsec:garbage_collector}
While poses are highly informative for action recognition, an input video
might contain irrelevant or idle zones, where the underlying poses are noisy
or non-discriminative to identify the actions being performed in the video. As
a result, low-scoring motion poselets could degrade the pose classifiers during
training, decreasing their performance. To deal with this problem, we include in
our model a \emph{garbage collector} mechanism for motion poselets. This
mechanism operates by assigning all low-scoring motion poselets to
the $(K+1)$-th pose dictionary entry. These collected poses are
associated with a learned score lower than $\theta^r$, as in Equation
(\ref{eq:motionposelets}). Our experiments show that this mechanism leads
to learning more discriminative motion poselet classifiers.

\input{learning}
\input{inference}

\subsection{Video Representation} \label{subsec:videorepresentation}

[EXPLAIN BETTER, ADD FIGURE]
Our model is based on skeleton information encoded in joint annotations.  We use the same geometric descriptor as in \cite{Lillo2014}, using angles between segments connecting two joints, and angles between these segments and a plane formed by three joints. In addition to geometry, other authors \cite{Zanfir2013,Tao2015,Wang2014} have noticed that including local motion information is beneficial to the categorization of videos. Moreover, in \cite{zhu2013fusing} the authors create a fused descriptor using spatio-temporal descriptors and joint descriptors, showing that they combined perform better than separated. With this is mind, we augment the original geometric descriptor with motion information: when there is only skeleton jonints data, we use the displacement of vectors (velocity) as a motion descriptor. If RGB video is available, we use the HOF descriptor extracted from the trajectory of the joint in a small temporal window.

For the geometric descriptor, we use 6 segments per human action (see Fig. XXXX). The descriptor is composed by the angles between the segments (15 angles), and the angles between a plane formed by three segments and the non-coplanar segments (3 angles). For motion descriptor, we use either the 3D velocity of every joint in each region as a concatenated vector (18 dimensions), or the concatenated HOF descriptor of the joint trajectories, transformed to a low-dimensional space using PCA (20 dimensions).