End-to-end learning of energy-based representations for irregularly-sampled signals and images

For numerous domains , including for instance earth observation , medical imaging , astrophysics , ... , available image and signal datasets often irregular spacetime sampling patterns and large missing data rates . These sampling properties is a critical issue to apply state-of-the-art learning-based ( e.g. , auto-encoders , CNNs , ... ) to fully benefit from the available large-scale observations and reach breakthroughs in the reconstruction and identification of processes of interest . In this paper , we address the end-to-end learning of representations of signals , images and image sequences from irregularly-sampled data , i.e . when the training data involved missing data . From an analogy to Bayesian formulation , we consider energy-based representations . Two energy forms are investigated : one derived from auto-encoders and one relating to Gibbs energies . The learning stage of these energy-based representations ( or priors ) involve a joint interpolation issue , which resorts to solving an energy minimization problem under observation constraints . Using a neural-network-based implementation of the considered energy forms , we can state an end-to-end learning scheme from irregularly-sampled data . We demonstrate the relevance of the proposed representations for different case-studies : namely , multivariate time series , 2 images and image sequences . 1 INTRODUCTION . In numerous application domains , the available observation datasets do not involve gap-free and regularly-gridded signals or images . The irregular-sampling may result both from the characteristics of the sensors and sampling strategy , e.g . considered orbits and swaths in spacebone earth observation and astrophysics , sampling schemes in medical imaging , as well as environmental conditions which may affect the sensor , e.g . atmospheric conditions and clouds for earth observation . A rich literature exists on interpolation for irregularly-sampled signals and images ( also referred to as inpainting in image processing ( 4 ) ) . A classic framework states the interpolation issue as the miminisation of an energy , which may be interpreted in a Bayesian framework . A variety of energy forms , including Markovian priors ( 12 ) , patch-based priors ( 20 ) , gradient norms in variational and/or PDE-based formulations ( 4 ) , Gaussian priors ( ) as well as dynamical priors in fluid dynamics ( 3 ) . The later relates to optimal interpolation and kriging ( 8 ) , which is among the state-of-the-art and operational schemes in geoscience ( 10 ) . Optimal schemes classically involve the inference of the considered covariance-based priors from irregularly-sampled data . This may however be at the expense of Gaussianity and linearity assumptions , which do not often apply for real signals and images . For the other types of energy forms , their parameterization are generally set a priori and not learnt from the data . Regarding more particularly data-driven and learning-based approaches , most previous works ( 2 ; 11 ; 20 ) have addressed the learning of interpolation schemes under the assumption that a representative gap-free dataset is available . This gap-free dataset may be the image itself ( 9 ; 20 ; 18 ) . For numerous application domains , as mentionned above , this assumption can not be fulfilled . Regarding recent advances in learning-based schemes , a variety of deep learning models , e.g . ( 7 ; 16 ; 24 ; 23 ) , have been proposed . Most of these works focus on learning an interpolator . One may however expect to learn not only an interpolator but also some representation of considered data , which may be of interest for other applications . In this respect , RBM models ( Restricted Boltzmann Machines ) ( 22 ; 6 ) are particularly appealing at the expense however of computationally-expensive MCMC schemes . In this paper , we aim to learn representations of signals or images from irregularly-sampled observation datasets . Our contribution is three-fold : • an end-to-end learning of energy-based representations from irregularly-sampled training data . Based on a neural-network architecture , it jointly embeds the considered energy form and an associated interpolation scheme . • besides classic auto-encoder representations , we introduce NN-based Gibbs-Energy representations , which relate to Markovian priors embedded in CNNs . • the demonstration of the relevance of the proposed end-to-end learning framework for different data types , namely time series , images and image sequences , with possibly very high missing data rates . The remainder is organized as follows . Section 2 formally states the considered issue . We introduce the proposed end-to-end learning scheme in Section 3 . We report numerical experiments in Section 4 and discuss our contribution with respect to related work in Section ? ? . 2 PROBLEM STATEMENT . In this section , we formally introduce the considered issue , namely the end-to-end learning of representations and interpolators from irregularly-sampled data . Within a classic Bayesian or energybased framework , interpolation issues may be stated as a minimization issue X̂ = arg min X Uθ ( X ) subject to XΩ = YΩ ( 1 ) where X is the considered signal , image or image series ( referred to hereafter as the hidden state ) , Y the observation data , only available on a subdomain Ω of the entire domain D , and Uθ ( ) the considered energy prior parameterized by θ . As briefly introduced above , a variety of energy priors have been proposed in the literature , e.g . ( 4 ; 20 ; 5 ) . We assume we are provided with a series of irregularly-sampled observations , that is to say a set { Y ( i ) , Ω ( i ) } i∈ { 1 , ... , N } , such that Ω ( i ) ⊂ D and Y ( i ) is only defined on subdomain Ω ( i ) . Assuming that all X ( i ) share some underlying energy representation Uθ ( ) , we may define the following operator I I ( Uθ , Y ( i ) , Ω ( i ) ) = arg min X Uθ ( X ) subject to XΩ ( i ) = Y ( i ) Ω ( i ) ( 2 ) such that I ( Y ( i ) , Ω ( i ) ) = X ( i ) . Here , we aim to learn the parameters θ ( ) of the energy Uθ ( ) from the available observation dataset { Y ( i ) , Ω ( i ) } i . Assuming operator I is known , this learning issue can be stated as the minimization of reconstruction error for the observed data θ̂ = arg min θ ∑ i ∥∥∥Y ( i ) − I ( Uθ , Y ( i ) , Ω ( i ) ) ∥∥∥2 Ω ( i ) ( 3 ) where ‖.‖2Ω refers to the L2 norm evaluated on subdomain . Learning energy Uθ ( ) from observation dataset { Y ( i ) , Ω ( i ) } i clearly involves a joint interpolation issue solved by operator I . Given this general formulation , the end-to-end learning issue comes to solve minimization ( 3 ) according to some given parameterization of energy Uθ ( ) . In ( 3 ) , interpolation operator I is clearly critical . In Section 3 , we investigate a neural-network implementation of this general framework , which embeds a neural-network formulations both for energy Uθ ( ) and interpolation operator I . 3 PROPOSED END-TO-END LEARNING FRAMEWORK . In this section , we detail the proposed neural-network-based implementation of the end-to-end formulation introduced in the previous section . We first present the considered paramaterizations for energy Uθ ( ) in ( 3 ) ( Section 3.1 ) . We derive associated NN-based interpolation operators I ( Section 3.2 ) and describe our overall NN architectures for the end-to-end learning of representations and interpolators from irregularly-sampled datasets ( Section 3.3 ) . 3.1 NN-BASED ENERGY FORMULATION . We first investigate NN-based energy representations based on auto-encoders ( 15 ) . Let us denote by φE and φD the encoding and decoding operators of an auto-encoder ( AE ) , which may comprise both dense auto-encoders ( AEs ) , convolutional AEs as well as recurrent AEs when dealing with time-related processes . The key feature of AEs is that the encoding operator φE maps the state X into a low-dimensional space . Auto-encoders are naturally associated with the following energy Uθ ( X ) = ‖X − φD ( φE ( X ) ) ‖2 ( 4 ) Minimizing ( 1 ) according to this energy amounts to retrieving the hidden state whose lowdimensional representation in the encoding space matches the observed data in the original decoded space . Here , parameters θ refer to the parameters of the encoder φE and decoder φD , respectively θE and θD . The mapping to lower-dimensional space may be regarded as a potential loss in the representation potential of the representation . Gibbs models provide an appealing framework for an alternative energy-based representation , with no such dimensionality reduction constraint . Gibbs models introduced in statistical physics have also been widely explored in computer vision and pattern recognition ( 13 ) from the 80s . Gibbs models relate to the decomposition of Uθ as a sum of potentials Uθ ( X ) = ∑ c∈C Vc ( Xc ) where C is a set of cliques , i.e . a set of interacting sites ( typically , local neighbors ) , and Vc the potential on clique c. In statistical physics , this formulation states the global energy of the system as the sum of local energies ( the potential over locally-interacting sites ) . Here , we focus on the following parameterization of the potential function Uθ ( X ) = ∑ s∈D ‖Xs − ψ ( XNs ) ‖2 ( 5 ) with Ns the set of neighbors of site s for the entire domain D and ψ a potential function . Lowenergy state for this energy refers to state X which operator ψ provides a good prediction at any site s knowing the state in the neighborhood Ns of s. This type of Gibbs energy relates to Gaussian Markov random fields , where the conditional likelihood at one site given its neighborhood follows a Gaussian distribution . We implement this type of Gibbs energy using the following NN-based parameterization of operator ψ : ψ ( X ) = ψ2 ( ψ1 ( X ) ) ( 6 ) It involves the composition of a space and/or time convolutional operator ψ1 and a coordinate-wise operator ψ2 . The convolutional kernel for operator ψ1 is such that the coefficients for the center of convolutional window are set to zero . This property fulfills the constraint that X ( s ) is not involved in the computation of ψ ( XNs ) at site s. As an example , for a univariate image , ψ1 can be set as a convolutional layer with NF filters with kernels of size 3x3x1 , such that for each kernel Kf Kf ( 1 , 1 , 0 ) = 0 ( the same applies to biases ) . In such a case , operator ψ2 would be a convolution layer with one filter with a kernel of size 1x1xNF . Both ψ1 and ψ2 can also involve non-linear activations . Without loss of generality , given this parameterization for operator ψ , we may rewrite energy Uθ as Uθ ( X ) = ‖X − ψ ( X ) ‖2 where ψ ( X ) ) at site s is given by ψ ( XNs ) . Overall , we may use the following common formulation for the two types of energy-based representation Uθ ( X ) = ‖X − ψ ( X ) ‖2 ( 7 ) They differ in the parameterization chosen for operator ψ .

Summary: The paper proposes a framework to learn representations of signals when they the signals are irregularly sampled.  They propose to do this by using some modified iteration steps from DINEOF algorithm. In addition to this, they propose a new energy function which is inspired by the idea of Gibbs distribution. The parametrize the energy function by some convolutional filters with a constraint. As claimed by the paper in the introduction, they only use the under sampled signals to learn as opposed to using the fully sampled ground truth signals in the previous deep learning basic approaches.