Discovery of Single Independent Latent Variable
Authors: Uri Shaham
Jonathan Svirsky
Ori Katz
Ronen Talmon
Venue: NeurIPS 2022 Accept (2022)
Keywords: Independent Component revcovery

Abstract: Latent variable discovery is a central problem in data analysis with a broad range of applications in applied science.
In this work, we consider data given as an invertible mixture of two statistically independent components, and assume that one of the components is observed while the other is hidden. Our goal is to recover the hidden component.
For this purpose, we propose an autoencoder equipped with a discriminator.
Unlike the standard nonlinear ICA problem, which was shown to be non-identifiable, in the  special case of ICA we consider here, we show that our approach can recover the component of interest up to entropy-preserving transformation.
We demonstrate the performance of the proposed approach in several tasks, including image synthesis, voice cloning, and fetal ECG extraction. 

Discovery of Single Independent Latent Variable
Uri Shaham
Jonathan Svirsky Faculty of Engineering Bar-Ilan University Ramat Gan, Israel svirskjebiu. ac
Department Computer Science
Bar-Ilan University
Ramat Gan, Israel
uri wwwhan@biu.scccmmmmooooooo ac. ac.
Ori Katz
Ronen Talmon
Electrical and Computer Engineering Technion
Electrical and Computer Engineering Technion Haifa, Israel ronenêee. technien. ac.
Haifa, Israel
orikats@canpus www.:uppp....ttttiii......c.
Abstract
Latent variable discovery is central problem in data analysis with broad range of applications in applied science. In this work, we consider data given as an invertible mixture of two statistically independent components, and assume that one of the components observed while the other is hidden. Our goal is to recover the hidden component. For this purpose, we propose an autoencoder equipped with discriminator. Unlike the standard nonlinear ICA problem, which was shown be -on-identifiable, in the special case of ICA we consider here, we show that our approach can recover the component of interest up to transformation. We demonstrate the performance of the proposed approach several tasks, including image synthesis, voice cloning, and fetal ECG extraction.
Introduction
The recovery of hidden components in data long-standing problem in applied science. This problem dates back to the classical PCA 44 20), yet it has numerous modern manifestations and extensions, e-g., kernel-based methods 47 source separation [T] B manifold learning [4 46 2
and latent Dirichlet allocation [4, to name but few. Perhaps the most relevant line of work in the context of this paper is independent component analysis (ICA) 23 which attempts decompose an observed mixture into statistically independent components.
Here, consider the following ICA-related recovery problem. Assume that the data is generated as an invertible mixture of two (not necessarily one dimensional) independent components, and that one the components is observed while the other is hidden. this setting, our goal is recover the latent component. At first glance, this problem setting may seem specific and perhaps artificial. However, posit that it fact broad and applies to many real-world problems.
For example, consider thorax and abdominal electrocardiogram (ECG) signals measured during labor for the purpose of determining the fetal heart activity. In analogy to problem setting, the abdominal signal can be viewed as a mixture of the maternal and fetal heart activities, the maternal signal be viewed activity is the hidden component of interest we wish to recover. In another example from a different domain, consider speech signal as mixture of independent components: the spoken text and the speaker identity. Arguably, the speaker identity is associated with the pitch and timbre, which are independent of information about the textual content, rhythm and volume. Consequently,
accurate proxy of the maternal heart activity alone, and the fetal heart
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

recovering a speaker-independen representation of the spoken text facilitates speech synthesis and
voice conversion.
In this paper, we present an autoencoder--asse approach, augmented with a discriminator, for this recovery problem. First, we theoretically show that this architecture of solution facilitates the recovery of the latent component up to an entropy-pres pnrrepy-presevving transformation. Second, in addition to the recovery of the latent component (the so-called analysis task), it enables to generate new mixtures corresponding to new instances of the observed independent component (the so-called synthesis task). Experimentally, we show both analysis and synthesis results on several datasets, consisting of simulated and real-world data. In particular, we demonstrate the proposed approach on ECG analysis, image synthesis, and voice cloning tasks.
Our contributions are as follows. (i) We propose an easy-to-train mechanism for extraction of a single latent independent component. (ii) We present simple proof for the ability of the proposed approach to recover the latent component. (iii) We experimentally demonstrate the applicability of the proposed approach in the contexts of both analysis and synthesis tasks. Specifically, we show applications to real-world data from different fields.
Related Work
The problem we consider in this work could be viewed as simplified case of the classical formulation of nonlinear ICA. Several algorithms have been proposed for recovery of the independent components, assuming that (i) the mixing of the components is linear, and (ii) the components (with a possible exception of one) are non- Gau-aaas; this case was proven to be identifiable [TO, 13. The nonlinear case, however, i.., when the mixing of the independent components is an arbitrary invertible function, was proven to be non-identifiable in the general case [24].
Identifiable nonlinear ICA. Hyvarinen et al. 25 have recently described general framework for identifiable nonlinear ICA, generalizing several earlier identifiability results for time series, e.g., [21 22 [5], in which in addition to observing the data the framework requires auxiliary observed variable u, so that conditioned on u, the latent factors are independent (i.c., in contrast to being marginally independent in standard ICA setting). The approach propose in this work falls into this general setting, as we assume that the auxiliary variable u is in fact one of the latent factors, which immediately satisfies the conditional independence requirement. For this special case we provide a simple and intuitive recovery guarantee.
Following works have recently extended the framework of Hyvarinen et al. 25 to generative models 33). unknown intrinsic problem dimension (50), and multiview setting 17 With respect to iVAE 30 we allow for the recovery of high-dimensional components, whereas in i'VAE, only one- dimensional components are recovered. In addition, our work presents several important differences: (i) we formulate our recovery guarantee result in terms of entropy prepeervigg map rather than statistical identifiability. (ii) It allows for a compact proof of the recovery guarantee. (iii) We present experiments on real-world data and comparisons to leading methods per application domain, whereas in (30, the iVAE approach was mostly demonstrated on simulated data.
Disentangled representation learning methods. While the main interest in ICA has originally been for purposes of analysis (i.e., recovery of the independent sources from the observed data), the highly impressive achievements in deep generative modeling in recent years have drawn much interest also to the direction of data synthesis (e.g., images) from independent factors. In the research community, this direction is often termed learning of disentangled representations, i.e., representations in which modification of a single latent coordinate in the representation affects the synthesized data by manipulating a single perceptual factor in the observed data, leaving other factors unchanged. a similar fashion to the ICA case, the task of learning disentangled representations in the general case was proved to be non-identifiable [39]. Several methods for learning disentangled representations have been recently proposed, most of which are based on a variational autoencoder (VAE, Kingma and Welling (32) formulation, and decompositions of the VAE objective, for example 19 31 33
[4] BAN-based approaches for disentanglement have been proposed as
well
8 5
Domain confusion. Our proposed approach based on the ability to learn an encoding in which different conditions (i.c., states of the observed factor) are indistinguishable. Such principle has been

---

Encoder E()
s'
Decoder D(.)
t
Reconstruction R
Discriminator
Independence I(,,)
g())
Figure 1: A diagram of the proposed approach. Learned functions colored in blue, while the objective functions are colored in gray.
in wide use in the machine learning community for domain adaptation tasks 15 to achieve this approximate invariance is via discriminator networks, for example, in 35 42 48 43 where the high-level mechanism is similar to the one proposed in this work, although the specifics are different.
popular means
We remark that while the algorithm we propose here has been presented before, e.g., in the context of singing voice conversion [3], to the best of our knowledge, it was not discussed in context of latent independent component recovery, and identifiability results were shown.
Problem Formulation
Let X Rd be random variable. We assume that X is generated via X f(, where f is an unknown invertible function two arguments, and S E S and T T are two random variables in arbitrary domains, satisfying S 11. T. We refer S as the unobserved source that wish to recover, to T as the observed condition, and to as the observed input.
Let ((s,, (:) be realizations of S T, and for all I..... let f(s; 44). Given only input-condition pairs, i.e., {(44, 4)) we state two goals. First, from analysis standpoint, we aim to recover the realizations 81,... 8n of the unobserved source S. Second, from synthesis standpoint, for any (new) realization of T,, we aim to to generate an instance f(,,, t) for any = 1,....,7. n.
4 Autoencoder Model with Discriminator
To achieve the above goals, we propose an autoencoder (AF) with discriminator. The AE model is denoted by (F, D), where the encoder F. maps inputs codes s' (analysis), and the decoder D maps code-condition tuples (s', () to input reconstructions å (synthesis). i.e., s' and (8', 22 ft The discriminator, denoted by g()), maps codes s' to predicted conditions
We two objective terms GAN-like minimax game:
min max [R (a. D(E()) XI (t.g((()))) E.D g
The first objective, denoted by R, measures the discrepancy between input and its reconstruction = t)) The second objective, denoted by I, quantifies the independence between the condition T and code S' via a prediction f = g(s'). In order to maximize -I (t, g (()))) the discriminator aims at leveraging any information in the code S' E(X) the condition T (via learned function g) obtain accurate prediction f g(s'). In order to minimize both R (x, (EE())(E)))))) t)) and -I (t, g (E(x))) the autoencoder aims at reconstructing the input from the code s' and the condition t, while failing the discriminator. We will show formally and empirically, that this results an equilibrium in which 5" does not contain any information on and contains all the remaining information in X. Our approach is illustrated in Figure
"This formulation of the discriminator does not capture all scenarios, however use it here for simplicity. In Section 4.2 we describe additional implementations of the discriminator.

---

4.1 Recovery of the Latent Component
Let S" = E be random variable representing the encoder output, and let X = D(S'.T) be random variable representing the decoder output.
Lemma 4. establishes that when the autoencoder is trained to perfect reconstruction and the learned code is independent of the condition, the learned code contains the same information S, thereby proving that the latent component of interest can be recovered, up to an preeopriign transfor- mation. The lemma is stated assuming is discrete, and in terms of mutual information ((s+)) and entropy H(Y) == in V. An equivalent result for the case of continuous S can be obtained by replacing the entropy term p(y) dy, where m(-) is with the limiting density of discrete points 27 HYY = Syey p(y) log mly)
Evey p(y) log p(y), where Y is a random variable with density p. taking values
the limiting density.
Lemma 4.1. Suppose we train the autoencoder model zero generalization loss, i.e. X X, and impose on the code that S' 1L T. Then 1 (S; = (()) = (S").
( T) HS, T). Since X ((S and ʃ is Proof. Since 11 T, H(S H(S
invertible, (S, T) = ((( Since X X, and since X is function S' , we have HX .)) = =H(X Therefore H(S]S') = 0, and - -(S))))) = (S))., Finally, since H(S") S H(S), we have H(S) = () H (()), hence H HS)
H( h(S" T) 0.
Lemma 4.1 has two important consequences. First, it shows that unlike the standard nonlinear ICA problem, the problem we consider here allows for recovery of the latent independent component of interest. More specifically, it proves that when the autoencoder yields perfect reconstruction and condition-nndependent code, the learned code is a recovery the random variable S, up to cntropy-preservingg transformation. Second, the lemma prescribes recipe for the practical solution we present here. Requiring the autoencoder to generate accurate reconstruction X of X ensures that no information on S is lost the encoding process. Independence of S' and is achieved implicitly; it results from the equilibrium of the GAN-like minimax game (I), as the discriminator can benefit from any mutual information between S' and T.
4.2 Training Objectives
As described above, to obtain code S' that is independent of the condition T, we utilize a discrimi- nator network, aiming to leverage information on T in S' for prediction, and train the encoder E to fail the discriminator, in a standard adversarial training fashion. Doing so pushes the learned codes S' towards being a condition-free encoding X.
We propose to optimize the following objectives of the discriminator and the autoencoder:
Laise min I (1, o( (F())))
(2)
CAB min (R (x, D(E(s), XI (L.))) F.D
The specific reconstruction and independence objective terms are application-dependent. our experiments, we make use of the following.
Reconstruction. We use standard reconstruction loss functions. In the experiments with images, L and SSIM loss (56] (and combinations of these) are used. é loss is also used in the audio experiments, and MSE loss in the experiments with ECG signals.
Independence. The discriminator computes map g(8),, where s' = Exx) is the code obtained from the encoder and = g(s') ) is the condition predicted by the discriminator. The discriminator is trained to minimize the independence term I(f. t) (thus to leverage mutual information in S' and T))
When the condition takes values from finite symbolic set. train the discriminator classifier that predicts the condition class from the code, and we set the independence term to I(Ê,t)

---

Cross Entropy t). This is also known as a Domain Confusion term. Using this term, the autoencoder trained to produce codes that maximize the cross entropy with respect the true condition, and the equilibrium of the game is when this term equals the cross entropy of a random guess.
When the condition and its prediction take numerical values, i.c., t, E R. we train the discriminator as a regression model and set the independence term to: t(f.t) Correll (f, t). We remark that this term also equals the negative R² term of a simple regression model, regressing t on f. Using this term, the autoencoder is trained to produce codes for which the squared correlation with the condition is minimized, As i is a nonlinear function of s' computed via flexible model such as neural net, (Co t))² implies that S' and T approximately statistically independent.
In addition, we also successfully train the discriminator in a contrastive fashion, i.c., to distinguish between "true tuples" (s', t) that correspond to samples (s, from the joint distribution of S and T satisfying s' = Exz) and x = f(s, t). and "fake tuples" (s',t), t), where s' = Exx) but x = f(, f) with & * f. The contrastive objective is then I(s', .) Cross Entropy(1, 1), where / is the ground truth true/fake label and g(s', t) is the predicted true/fake label made by the discriminator. Note that this implementation deviates from the discriminator formulation we used thus far. Here, the discriminator -) maps tuples of codes and conditions (s', .t) true/fake labels.
We remark that other possible implementations of the independence criterion can be utilized as well, e.g., nonlinear CCA
[ 41 and the Hfllbert-Schmic Independence Criterion (HSIC) (18).
Optimizers. Our proposed approach utilizes two optimizers, one for the autoencoder and one for the discriminator. The AE optimizer optimizes LAE by tuning the encoder and decoder weights. The discriminator optimizer optimizes Luse by tuning the discriminator weights (which determine the function g). A common practice in training GANs is to call the two optimizers with different frequencies. We specify the specific choices used in our experiments in Appendix ??.
GAN real/ fake discriminator. Optionally, a GAN-like real / fake discriminator can be added as an additional discriminator in order to encourage generating more realistic inputs. While we have a successful empirical experience with such GAN discriminators (e.g., see Appendix ??), this is not core requirement of our proposed approach.
Experimental Results
In this section, demonstrate the efficacy of the proposed approach in various settings, by reporting experimental results obtained on different data modalities and condition types, in both analysis and synthesis tasks. We present four applications here and additional two in the appendix.
We begin with a two dimensional analysis demonstration, in which the condition is real-valued. Second, we demonstrate the utility of our approach for image manipulation, where the condition given as an image. Third, the proposed approach is used for voice cloning, which is primarily synthesis task with a symbolic condition. Fourth, apply our approach to ECG analysis task, using a real-valued heartbeat signal as the condition. Additional experimental results in image synthesis are described Appendix ?? and ??.
The network architectures and training hyperparameters used in each of the experiments are described Appendix ??. In addition, codes reproducing some of the results in this manuscript are available at hhttts::// : //github.tttttt/// com/ shaham Shahammlab/disst lab/disilv
5.1 2D Analysis Demonstration
In this example, we first generate the latent representation of the data by sampling from two indepen- dent uniform random variables. We then generate the observed data via linear mixing. We consider one of the latent components as the condition and train the autoencoder to reconstruct the observed data, while obtaining code which is independent of the condition using the regression objective. We use & as a reconstruction term. The top row in Figure 2 shows the latent, observed and reconstructed data, as well as the distribution of the condition and the learned code. The bottom row in Figure 2 shows the results of similar setup, except for the mixing which is now nonlinear. As can be seen, the joint distribution of the learned code and the condition is approximately a tensor product of the marginal distributions, which implies that the latent component is indeed recovered.
5

---

(5).t
Observed
Reconstructio
vs. ($'.)
Unear mixing
Noriinear miking
Figure 2: Analysis demonstration. The reconstruction plots show train data in red and reconstructed test data in blue. The learned code s' is recovery of the latent factor s, up transformation (c.g., an arbitrary monotonic transformation). The approximate independence of the S and the condition T can be recognized by noticing that the joint density of the code and the condition is an outer product of the marginal distributions. The p-values of x² independence test for the shown results 0.75 (linear mixing) and 0.83 (nonlinear mixing)
5.2 Rotating Figures
In this experiment, we use the setup shown in Figure 3 in which two figures, Bulldog and Bunny, rotate on dises. The rotation speeds are different and figures are recorded by two static cameras, where the right camera captures both Bunny and Bulldog, while the left camera captures only Bulldog. The cameras operate simultaneously, so that in each pair of images Bulldog's position with respect to the table is the same. This dataset was curated in [36).
integer multiple one of the other. The
We consider images from the right camera (which contain both figures) the observed input and the images from the left camera (which only show Bulldog) as the condition t. Note that the input can be considered generated from two independent sources, namely the rotation angles of Bulldog and Bunny. The goal is to use x and to recover the rotation angle of Bunny
Once training is done, we use the autoencoder to generate new images by manipulating Bulldog's the encoder, obtaining an
rotation angle while preserving Bunny's. This is done by feeding encoding s', sampling arbitrary condition i and feeding (s', i) through the decoder. We use E loss for reconstruction, and contrastive loss to train the discriminator. Namely, we train the discriminator to distinguish between (image, condition) tuples, which were shot at the same time, and tuples which were not. Figure | shows an exemplifying result. As can be seen, the learned model disentangles the rotation angles of Bunny and Bulldog and generates images in which Bunny's 's rotation angle is preserved while Bulldog's is manipulated.
5.3 Voice Cloning
To demonstrate the application of the proposed method to voice conversion, we run experiments on non-pun non-parallel corpus, CSTR VCTK Corpus [53] which consists of 109 English speakers with several accents (e.g., English, American, Scottish, Irish, Indian, etc.). In our experiments, we use subset of the corpus containing all the utterances for the first 30 speakers (p225- p256, without p235 and p242).
We construct the autoencoder to operate on Mel spectrograms using the speaker id as the condition. The AE architecture was based Jasper (38) blocks (specific details can be found in Appendix ??). The decoder uses a learnable lookup table with 64-dimensional embedding for each speaker. For the discriminator, use the same architecture as in 42). We use E loss for reconstruction, and the discriminator is trained using domain confusion loss. Along with the reconstruction loss, in this
A related work on this dataset was done in 49 although there the goal was the opposite one, i.c. to recover the common information of the views, which is the rotation angle of Bulldog.

---

Figure 3: Experiment setup of the rotating figures. Buildog and Bunny rotate in different speeds. The right camera captures both Buildog and Bunny, while the left camera captures only Bulldog. Images from the right camera are considered as the input which is generated from two independent factors the rotation angles of the figures. Bulldog considered as the condition The goal is to recover the rotation angle of Bunny, and manipulate a given input image by plugging in different condition than the one present the image. The fact that t and æ are captured from two different viewpoints prevents modification of the image simply by pasting Bulldog into Æ.
org image
condition
reconstruction
new condition
synthesis
Figure 4: The rotating figures experiment. From left to right: (i) the input image to the encoder æ, (ii) the condition corresponding to Bulldog in z as captured from the viewpoint of the left camera, (iii) the reconstruction of x, (iv) new condition plugged into the decoder, and (v) the resulting manipulated image.
experiment we also train a real/fake discriminator, applied to the output of the decoder. To convert the decoder output waveform, we pre-trained melgan 34] vocoder.
Once the autoencoder is trained, we apply convert speech from any speaker to any other speaker. Some samples of the converted speech are given at lhttps://shaham-bb lab. github. to/disilv/ To evaluate the similarity of the converted voice and the target voice, we use MCD (Mel Cepstral Distortion) on subset of the data containing parallel sentences of multiple speakers. Specifically, MCD computes the & difference between dynamically time warped instance of the converted source voice and a parallel instance of the target voice, which is common evaluation metric for voice cloning. We remark that the parallel data are used only for evaluation and not for training the model. We use the script provided in 37 compute the MCD and compare proposed approach to 12 45 and references therein, which are all considered to be strong baselines, trained on the VCTK dataset well. The results, shown in Table I demonstrate that approach outperforms these strong baselines.
5.4 Fetal ECG extraction
In this experiment, demonstrate the applicability of the proposed approach non-invasive fetal electrocardiogram (fECG) extraction, which facilitates the important task of monitoring the fetal cardiac activity during pregmancy and labor. Following commonly commonly-used non non-invasive methods, we consider extraction of the fECG based on two signals: (i) multi-channel abdominal ECG recordings, which consist of mixture of the desired fECG and the masking maternal electrocardiogram (mECG), and (ii) thorax ECG recordings, which are assumed to contain only the mFCG. In analogy to our problem formulation (see Section the desired unobserved source s denotes the fFCG, the observed condition & denotes the (thorax) mECG, and the input æ denotes the abdominal ECG.
7

---

Table 1: Voice cloning results: Mel Cepstral Distortion (MCD) in terms of mean (std). PPG, PPG2
results are taken from 45 VQ AVE and PPG GMM results are taken from [2]
METHOD
TTS SKINS
45
GLE
12
VQ VAE
PPG GMM
PPG
PPG2
OURS
MCD
8.76 (1.72)
7.56
8.43
8.57
9.19 (1,50)
9.18 (1.52)
6.27 (1.44)
Table fECG extraction results. In the leftmost column, we present Rg, and in the other columns we present Ry achieved by the different methods.
# OF SUBIECT
INPUT
OURS
ADALINE
ESN
LMS
R
TOP 5
2.23 (3.23)
6.86 (1.98)
6.46 (2.54)
1.99 (1.08)
2.60 (1.60)
1.03 (0.70)
ToP 10
1.20 (2.41)
(2.02)
4.22 (2.94)
1.19 (1.10)
1.56 (1.53)
0.75 (0.56)
TOP 20
0.66 (1.75)
(2.44)
2.59 (2.63)
0.71 (0.91)
0.89 (1.26)
0.51 (0.46)
ALL
0.30 (1.17)
(2.16)
1.32 (2.08)
0.36 (0.68)
0.40 (U.94)
0.26 (0.38)
Dataset. We consider the dataset from 122 which is publicty available on PhysioNet [6]. This dataset was recently published and is part of an ongoing effort establish a benchmark for non- invasive fECG extraction methods. The dataset consists of ECG recordings from 60 subjects. Each recording consists of nh 24 abdominal ECG channels and n, thorax ECG channels. In addition, it contains a wals doppler recording of the fetal heart that serves as a ground-truth. See Appendix ?? for more details.
Model training. The input-condition pairs (2,, t,) are seme-segments of the abdominal ECG record- ings (I; E IR tR. x*T) ) and the thorax ECG recordings (4, E R x**), where the length of the time-segments is set to ny = 2, 000 (4 seconds). We train a separate model for each subject based on collection of input-condition pairs ((x;, 4)) of time-segments.
The encoder is based on a convolutional neural network (CNN), so that the obtained codes sf = E(x;) RR XrT are time-segments, where the dimension of the code is set na 5. For more details on the architecture, model training, and hyperparameters selection, see Appendix ??.
We note that the training is performed in an unsupervised manner, i.e., we use the ground-truth doppler signal only for evaluation and not during training.
Qualitative evaluation. In Figure 5 we present an example of an input-condition pair (x,, 4)) and the obtained code s/ F( 4.). We see that the abdominal channels consist of a mixture of the FECG and the mECG, where the FECG is significantly less dominant than the mECG and might even be completely absent from some of the channels. In addition, we see that the thorax channels are affected by the mECG only. Lastly, we see that the obtained code captures the IECG without any noticeable trace of the mECG. In addition, we present the projections of 1, 000 sequentially-samppl inputs (abdominal channels), conditions t; (thorax channels), and their codes sf Exx;) on their respective principal components. We color the projected points by the periodicity of the mECG (middle column), computed from the thorax channels, and by the periodicity of the fECG (rightmost column), computed from the ground-truth -rrth doppler signals. We see that the PCA of the abdominal and thorax channels are similar, implying that the mECG dominates the mixture. In addition, we see that the color of the PCA of the abdominal and thorax channels according to the mECG (middle column) is similar and smooth, unlike the color by the fECG (rightmost column). In contrast, the PCA of the code different (bottom row) and only the color by the fECG is smooth (rightmost column), indicating that the code captures the IECG without significant trace of the mECG, as desired.
Baselines. We consider four baselines taken from recent review methods that utilize reference thorax channels. The first two baselines are based on adaptive filtering. which is considered be the traditional approach for fECG extraction: least mean squares (LMS) and recursive least squares (RLS). This approach was first introduced by Widrow et al. (57). and it
Specifically, we focus on
still considered to be relevant recent studies 40 58 53). The third baseline is ADALINE 22
which utilizes neural networks adaptable to the nonlinear varying properties of the ECG signal.
The fourth baseline is based on an echo state network (ESN) 26
8

---

Abdbmvi Channes
Thoia
Figure 5: Example of an input-condition pair (24,4) and the obtained code s/. The duration of the presented time segment is sec. Leftmost column: the raw channels. Middle column: the PCA embedding of the samples colored by the mECG. Rightmost column: the PCA embedding of the samples colored by the fECG. Top row: the abdominal channels z; (for brevity only channels are presented). Middle row: the thorax channels 4, Bottom row: the obtained code 6.
Quantitative evaluation. To the best of our knowledge, there no gold-standard nor definitive evaluation metrics for IECG extraction. Here, based on the ground-truth doppler signal, we quantify the enhancement of the IECG and the suppression of the mECG as follows.
First, we compute the principal component of the input 24. Second, we compute the one-sided auto- correlation of the principal component, and denote it by Az,- Then, quantify the average presence of the fECG in the inputs I; by computing: A
Σ Az, (7, where -. denotes the periods of the fECG obtained from the doppler signals, and ni denotes the number of time segments in the evaluated recording. Similarly, we compute A n. E (T, (T."))), where T. (m) denotes the periods of the mECG obtained from the thorax signals. Finally, to quantify the relative presences ACC
of the signals, we compute the ratio R2 = A!m! We apply the same procedure to the codes s/,, resulting in Rg. When evaluating the baselines, we consider the signals obtained after the mECG cancellation as the counterparts of our code signals.
In Table 2 present the average ratios in the input, code, and baselines over all the subjects (see Appendix ?? for results per subject). We note that not all the subjects in the dataset include noticeable fECG in the abdominal recordings. Therefore, subjects showing highest average ratios Rg. We see that our method significantly enhances the fECG with respect to the mixture, and it outperforms the tested basclines.
present results over subsets of top
6 Conclusion
In this paper, we present autoencoder-based approach for single independent component recovery. The considered problem consists of observed data (mixture) generated from two independent compo- nents: one observed and the other hidden that needs to be recovered. We theoretically show that this ICA-related recovery problem can be accurately solved, in the sense that the hidden component is recovered up to entropy-preserring preserving function, by autocncoder equipped with discriminator. In addition, we demonstrate the relevance of the problem and the performance of the proposed solution on several tasks, involving image manipulation, voice cloning, and fetal ECG extraction.
Future research will address the limitations of this work. Lemma 4.1 assumes zero generalization loss, ie., convergence to global minimum, which is often not achieved in practice. In future work we plan to generalize this statement and assume bounded generalization loss. Another future direction will address noise robustness. The current setting does not consider any noise, either in / or in or even in f. For example, the presence of noise, perfect reconstruction undesired, and other losses need to be developed and used.

---

Acknowledgments and Disclosure of Funding
We thank the reviewers for their important comments and suggestions. The work of OK and RT was supported by the European Union's 's Horizon 2020 research and innovation programme under grant agreement No. 802735-ERC-DIEE Advancement Chair in AL.
RT acknowledges the support of the Schmidt Career
References
[1] Andrew, G., Arora, R., Bilmes, J., and Livescu, K. (2013). Deep canonical correlation analysis. International conference on machine learning, pages 1247-1255. PMLR.
[2] Belkin, M. and Niyogi, P. (2003). Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373-1399
[3] Belouchrani, A., Abed-Meraim, K., Cardoso, J.-F., and Moulines, E. (1997). A blind source separation technique using second-order statistics. JEEE Transactions on signal processing, 45(2):434-44) 444.
[4] Blei, M., Ng, A. Y., and Jordan, M. (2003). Latent dirichlet allocation. the Journal of machine Learning research, 3:923-1022.
[5] Brakel, P. and Bengio. Y. (2017). Learning independent features with adversarial nets for non-linear ICA. arXiv preprint arXiv:1710.05050.
[6] Burgess, C. P., Higgins, I., Pal, A., Matthey, L.., Watters, N., Desjardins, G., and Lerchner, A. (2018). Understanding disentangling in beta-VAE. arXiv preprint
[7] Chen, R. T., Li, X., Grosse, B., and Duvenaud, D. K. (2018). Isolating sources of disen- tanglement in variational autoencoders. Advances in neural information processing systems, 31.
[8] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. (2016). Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29.
[9] Coifman, R. R. and Lafon, S. (2006). Diffusion maps. Applied and computational harmonic analysis, 21(1):5-30.
[10] Comon, (1994). Independent component analysis, a new concept? Signal processing. 36(3):287-314.
[11] Comon, P. and Jutten, C. (2010). Handbook of Blind Source Separation: Independent component analysis and applications. Academic press.
[12] Ding, S. and Gutierrez-Osuna, R. (2019). Group latent embedding for vector quantized varia- tional autoencoder in non-parallel voice conversion. INTERSPEECH, pages 724-728.
[13] Eriksson, J. and Koivunen, (2004). Identifiability, separability, and uniqueness of linear ICA models. IEEE signal processing letters, 11(7):601-60
[14] Esmacili, B.. Wu, H., Jain, S., Bozkurt, A., Siddharth, N., Paige, B., Brooks, D. H., Dy, J., and Meent, J.-W. (2019). Structured disentangled representations. In The 22nd International Conference Artificial Intelligence and Statistics, pages 2525-2534 PMLR.
[15] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. (2016). Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096-2300.
[16] Goldberger, L.,, Amaral, A., Glass, L., Hausdorff. M., Ivanov, C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., and Stanley, H. E. (2000). Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation. 101(23)ee11-<2220.
10

---

[17] Gresele, L., Rubenstein, P. K., Mehrjou, A., Locatello, F., and Schölkopf, B. (2020). The incom- plete rosetta stone problem: Identifiability results for multi-view nonlinear ICA. In Uncertainty in Artificial Intelligence, pages 217-227. PMLR.
[18] Gretton, A., Bousquet, O., Smola, A., and Schölkopf, B. (2005). Measuring statistical depen- dence with Hilbert-Schmidt norms. In International conference on algorithmic learning theory, pages 63-77. Springer.
[19] Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2016). Beta- VAE: Learning basic visual concepts with a constrained variational framework.
[20] Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of educational psychology. 24(6):417.
[21] Hyvarinen, A. and Morioka, H. (2016). Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in neural information processing systems, 29.
[22] Hyvarinen, A. and Morioka, H. (2017). Nonlinear ICA of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469. PMLR.
[23] Hyvärinen, A. and Oja, (2000). Independent component analysis: algorithms and applications. Neural networks, 13(4-5):111433)
[24] Hyvärinen, A. and Pajunen, P. (1999). Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-433
[25] Hyvarinen, A., Sasaki, H., and Turner, R. (2019). Nonlinear ICA using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR.
[26] Jaeger, H. (2001). The "echo state" approach analysing and training recurrent neural networks- with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148(34):13.
[27] Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical review, 106(4):620.
[28] Kahankova, R., Martinek, R., Jaros, R., Behbehani, K., Matonia, A., Jezewski, M., and Behar, J. A. (2019). A review of signal processing techniques for non-invasive fetal electrocardiograply, IEEE reviews biomedical engineering, 13:51-73.
[29] Kahankova, R., Martinek, R., Mikolášová, M., and Jaroš, R. (2018). Adaptive linear neuron for fetal electrocardiogram extraction. In 2018 IEEE 20th International Conference e-Heaalth Networking, Applications and Services (Healihhomm), pages 1-5, IEEE
[30] Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A. (2020). Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217 PMLR.
[31] Kim, and Mnih, A. (2018). Disentangling by factorising. In International Conference on Machine Learning, pages 2658. PMLR.
[32] Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. International Conference on Learning Representations (ICLR).
[33] Kumar, A., Sattigeri, P., and Balakrishnan, A. (2018). Variational inference of disentangled latent concepts from unlabeled observations. In International Conference Learning Representations.
[34] Kumar, K., Kumar, R., de Boissiere, T., Gestin, L.,, Teoh, W. 7., Sotelo, J., de Brébisson, A., Bengio, Y., and Courville, A. C. (2019). Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32.
[35] Lample, G., Zeghidour, N., Usunier, N., Bordes, A., Denoyer, L.., and Ranzato, M. (2017). Fader networks: Manipulating images by sliding attributes. Advances neural information processing systems, 30.
11

---

[36] Lederman, R. R. and Talmon, R. (2018). Learning the geometry of common latent variables using alternating diffusion.
-uteraain-diifuusion. Applied and Computational Harmonic Analysis, 44(3):509-5366
[37] Li, C., Shi, J., Zhang. W., Subramanian, A. S., Chang, X., Kamo, N., Hira, M., Hayashi, T., Boeddeker, C., Chen, Z., and Watanabe, S. (2021). ESPnet-SE: End-to-end speech enhancement and separation toolkit designed for ASR integration. In Proceedings of IEEE Spoken Language Technology Workshop (SI.T), pages 785-792. IEEE.
[38] Li, J., Lavrukhin, V., Ginsburg. B., Leary, R., Kuchaiev, O., Cohen, J. M., Nguyen, H., and Gadde, R. T. (2019). Jasper: An end-to-end convolutional neural acoustic model. Proc. Interspeech.
[39] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14-4124. PMLR.
[40] Martinek, R., Kahankova, R., Nazeran, H., Koneeny, J., Jezewski, J., Janku, P., Bilik, P., Zidek, J., Nedoma, J., and Fajkus, M. (2017). Non-invasive fetal monitoring: A maternal surface ECG electrode placement- bbased novel approach for optimization of adaptive filter control parameters using the LMS and RLS algorithms. Sensors, 17(5):155.
[41] Michaeli, T., Wang, W., and Livescu, K. (2016). Nonparametric canonical correlation analysis. In International conference on machine learning, pages 1967-1976. PMLR.
[42] Mor, N., Wolf, L.., Polyak, A., and Taigman, Y. (2019). A universal music translation network. International Conference on Learning Representations (ICLR).
[43] Nachmani, E. and Wolf, L.. (2019). Unsupervised singing voice conversion. Proc. Interspeech.
[44] Pearson, K. (1901). Liii on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559-572.
[45] Polyak, A., Wolf, L., and Taigman, Y. (2020). TTS skins: Speaker conversion via ASR. Proc. Interspeech.
[46] Roweis, S. T. and Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2333-2533 2326.
[47] Schölkopf. B., Smola, A., and Müller, K.-R. (1998). Nonlinear component analysis as a kemel eigenvalue problem. Neural computation, 10(5):1299-1311)
[48] Shaham, U. (2018). Batch effect removal via batch-free encoding. bioRxiv, page 380816.
[49] Shaham, U. and Lederman, R. R. (2018). Learning by coincidence: Siamese networks and common variable learning. Pattern Recognition, 74:52-63.
[50] Sorrenson, P., Rother, C., and Köthe, U. (2020). Disentanglement by nonlinear ICA with general incompressible-fiow networks (GIN). arXiv preprini a4X4422001.04872.
[51] Sprekeler, H., Zito, T., and Wiskott, L. (2014). An extension of slow feature analysis for nonlinear blind source separation. The Journal of Machine Learning Research, 15(1):921-947.
[52] Sulas, E., Urru, M., Tumbarello, R., Raffo, L.,, Sameni, R., and Pani, D. (2021). A non- invasive multimodal foetal ECG-doppler dataset for antenatal cardiology research. Scientific Data, 8(1):1-19.
[53] Swarnalath, R. and Prasad, D. (2010). A novel technique for extraction of FECG using multi stage adaptive filtering. Journal of Applied Sciences, 10(4):319-324
[54] Tenenhaum, J. B., De Silva, V., and nonlinear dimensionality reduction. Science, 290(5500)2219--3233
and Langford, J. C. (2000). A global geometric framework for
[55] Veaux, C., Yamagishi, J., MacDonald, K., et al. (2017). CSTR VCTK corpus: English multi- speaker corpus for CSTR voice cloning tooliit. University of Edinburgh. The Centre for Speech Technology Research (CSTR).
12

---

[56] Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612.
[57] Widrow, B., Glover, J. R., McCool, J. M., Kaunitz, J., Williams, C. S., Hearn, R. H., Zeidler, J. R., Dong, J. E., and Goodlin, R. C. (1975). Adaptive noise cancelling; Principles and applications. Proceedings of the IEEE, 63(12): 1692--1666.
[58] Wu, S., Shen, Y., Zhou, Z.,, Lin, L.,, Zeng, Y., and Gao, X. (2013). Research of fetal ECG extraction using wavelet analysis and adaptive filtering. Computers in biology and medicine, 13210):1622-1627
13


Reviews:
[{"review_id": "w0s9g8sEMDq", "reviewer": "Reviewer_kukj", "summary": "This work studies a method to perform non linear ICA where the observed signals $x$ are a non-linear mixing of two independent variables, the sources $s$ and the observed condition $t$: $x = f(s, t)$. The goal of the problem is to recover $s$ and a way to generate new samples $x$ by providing only samples of $x$ and $t$.\n\nTo solve the problem, the authors consider an auto-encoder architecture, where the encoder should output the sources $s$ given $x$ as input, the decoder takes the estimated source of the encoder and the condition $t$ as input, and tries to reconstruct the input $x$. A discriminator is also used, to enforce independence between the condition and the estimated sources. Training is performed by solving a min-max optimization problem, where the discriminator tries to minimize independence between the output of the encoder and the condition, while the encoder and decoder are trained to maximize a weighted sum of independence and reconstruction.\n\nThe authors demonstrate that if the signals are perfectly recovered by the AE and if the sources $s'$ output by the encoder are independent from T, then $s$ and $s'$ are identical up to an entropy-preserving transform.\n\nThe authors then demonstrate the usefulness of the framework on a variety of tasks: synthetic examples, a computer vision disentanglement example, a voice cloning experiment, and a fetal ECG extraction problem.", "questions": "- In lemma 4.1, wouldn't it be clearer to write the conclusion as $h(S|S') = 0$? \n", "limitations": "I cannot see potential negative societal impact with this work.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-------------\n- This paper is extremely well written and pleasant to read. I have learned a lot by reading it.\n- The framework proposed is sound and seems novel (even though I am far from an expert on non-linear ICA)\n- Theory seems sound\n- The experiments are very convincing and clearly demonstrate the wide applicability of the method. They do a great work to show that these ideas go way beyond what was first proposed in [45] for voice conversion.\n\nWeaknesses\n-----------------\n\n- The authors could more clearly delineate their work, especially regarding [45], e.g. it should be made clearer that both the architecture in Fig1 and the training objective come from this paper (I find the sentence \"We remark that while the algorithm we propose here has been presented before\" quite vague)\n- The ground truth doppler recording could be displayed in fig.5.\n\n\nMisc: typo L 218 \"lossto\" ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RNlBHbTF5wU", "reviewer": "Reviewer_gAAc", "summary": "This paper introduces a novel technique for the extraction of latent variable (or hidden source) form a mixture of independent components. The work is a special case of ICA, extending on the nonlinearity and also offering an identifiable solution. The authors highlight the capabilities on four different applications, including image analysis (synthesis), voice cloning and biomedical data analysis.", "questions": "The proposed approach was only evaluated qualitatively and no quantitative evaluation were provided for any of the applications. Could the authors comment why they did not use any quantitative measure (image similarity measure, quality of fetal heart rate monitoring for the fetal ECG application)?\nThe authors claim that there is no gold-standard for the evaluation metrics for fECG extraction. But that was the purpose of the 2013 PhysioNet/Computing in Cardiology challenge, which proposed metrics for the quality of the fetal heart rate monitoring. The authors could have assessed how the extracted source (fECG) was reliable for estimating the fetal heart rate and if it was in accordance was the fHR estimated from the Doppler signal. \nI would also refer the authors to the review Behar, et al. (2016). A practical guide to non-invasive foetal electrocardiogram extraction and analysis. Physiological measurement, 37(5), R1. This review present methods and metrics to be used for the evaluation of fECG extraction techniques.\nThe interpretation of figures 3, 4, 5 is also really difficult, and readers might struggle to understand the takeaway messages of these figures. For example, the explanation of the qualitative estimation of the fECG problem is given in annex C.6.4, which makes it very hard to understand and interpret the figure 5.\nCould the authors comment on how their techniques are coping with noisy measurements, as it seems most of the database used have relatively low level of noise.\nCould the authors discuss the impact of the choice of the encoder decoder part of the proposed approach. They suggested the use of a ResNet for image analysis, and simple CNN for fECG analysis, how did they make these decisions.\nThe caption of the figures should be self-explanatory, (for example figure 1 remind the reader what the different variables stand for (x, t, s’…). \nSome acronyms have been introduced without being defined beforehand. For example, line 64, iVAE was introduced without explanations, whereas VAE was introduced and explained on line 82.\n", "limitations": "The potential negative impact of the voice cloning applications are discussed, but this discussion is relegated to the annex, which I believe not to be appropriate.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: (i) The proposed approach has been tested on several applications, ranging form image analysis, voice cloning, and biomedical data analysis.\n(ii) The proposed approach is quite original (to my knowledge), and the use of a discriminator is quite interesting for the extraction of a “disentangled” representation.\nWeaknesses: (i) The authors have not evaluated their approach quantitively, and compared their technique with other SOFA techniques. The use of already suggested quantitative metrics is possible for FECG application, and would have been useful.\n(ii) The authors did not perform an ablation study, and have not highlighted the added-vale of the discriminator part for analysis purpose. It would have been great to assess a simple VAE for fECG extraction, using the same network architecture to highlight the higher performance introduced by the use of the discriminator.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I6bxeB_45lE", "reviewer": "Reviewer_NayA", "summary": "This paper studies a special case of non-linear ICA where one of the latent source is known and the goal is to recover the other. \nWhile identifiability results are already known in this setting [27], this paper formulates its guarantees thanks to an entropy-preserving map rather than statistical identifiability and comes with multiple experiments on synthetic and real data (from very different fields).", "questions": "- Identifiability: How does the identifiability result compare with [27] ? It seems to me that statistical identifiability is a stronger result than entropy preserving maps. Could your results be derived from [27] by letting the sources by high dimensional ?\n\n\n- Unclear paragraph: \n(l163): \"Correl(ˆt, t))2 = 0 implies that S′ and T are approximately statistically independent\". In what sense are they approximately statistically independent ? Under which measure of independence ?\n\n(l164-170): Could you explain why training the discriminator in a contrastive fashion leads to minimization of the mutual information between S' and T ? I also believe there is a typo l166 where you wrote $x = (s, \\tilde{t})$ I believe it should be $x = f(s, \\tilde{t})$. I believe I just need more details to understand this part.\n\n- Other concerns:\n In the legend of Figure 2. Using chi square test this way is wrong, it can only be used to reject independence, not to prove it. ", "limitations": "Ethical limitations are properly addressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The paper is overall very well written. The high level description is very clear, the goals are well described and well motivated.\n\n- The large number of experiments on data from different field shows that the problem is indeed of interest.\n\n- The results are overall convincing\n\nWeaknesses:\n\n- The novelty of the identifiability result is questionable (see the question section) \n\n- l162 -> 170 are a bit unclear (see the question section)\n\n- I tried to open the notebook notebooks/2D_demo.ipynb and it failed with the error \nUnreadable Notebook: notebooks/2D_demo.ipynb NotJSONError(\"Notebook does not appear to be JSON: ''...\")\n\nThen, I opened the ECG folder and run Main.py. The code seems to run but with worrying warnings: \npadasip/filters/rls.py:183: RuntimeWarning: overflow encountered in matmul\n  R1 = self.R @ (x[:, None] * x[None, :]) @ self.R\npadasip/filters/rls.py:183: RuntimeWarning: invalid value encountered in matmul\n  R1 = self.R @ (x[:, None] * x[None, :]) @ self.R\n<class 'ValueError'>\n(\"Input contains NaN, infinity or a value too large for dtype('float64').\",)\nInput contains NaN, infinity or a value too large for dtype('float64').\nIn general, the code quality contrasts strongly with the quality of the paper. It does not follow most of the coding guidelines in Python and in particular it comes without any documentation or unit tests. The lack of a clear API prevents easy reproducibility.\n\nIn order for others to be able to re-use your work, it would be nice to correct at least the 2D-demo and make it a proper python package with tests, clear documentation, examples (the 2D-demo comes as a perfect example), continuous integration (although I understand this last one is difficult to include in a submission but should definitely be there after publication). It does not take that much time to do and would definitely help other researchers build upon your work.\n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ChGRk3dgg8U", "reviewer": "Reviewer_nS8k", "summary": "This paper studies the problem of nonlinear independent component analysis in a special case where there are only two components and one is observed. The authors proposed an autoencoder model with a GAN-style additional regularization term which encourages the latent component to be independent to the observed one. The authors proved that the proposed method can recover the latent component up to entropy-preserving transformation. The performance of the proposed method is demonstrated on real world data.\n", "questions": "1. The coefficient for the independence loss lambda takes a different value for each experiment. How is it tuned? Are the results sensitive to the values of lambda?\n2. The performance of the proposed method is evaluated in terms of the quality of reconstruction of the same hidden component with a new observed component. This task could also be solved by content-style factorization approaches, although content-style factorization approaches do not force the hidden component to be independent. It would be nice to compare the proposed method with content-style factorization methods.\n", "limitations": "The proof only covers the ideal scenario in which the autoencoder model has zero generalization error, which is impractical.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- A new autoencoder-based formulation for discovering one hidden component with nonlinear mixing.\n- A theoretical guarantee for the discovery of the hidden component\n- The proposed method is evaluated on real world data\n\nWeaknesses:\n- The proposed method can only be applied to a restricted nonlinear ICA setting. It can only handle two components, in which one needs to be observed.\n- The proof only covers the ideal scenario in which the autoencoder model has zero generalization error, which is impractical. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept