Geometrically Aligned Transfer Encoder for Inductive Transfer in Regression Tasks
Authors: Sung Moon Ko
Sumin Lee
Dae-Woong Jeong
Woohyung Lim
Sehui Han
Venue: ICLR 2024 poster (2024)
Keywords: Transfer Learning
Inductive Transfer
Geometrical Deeplearning
Regression
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A novel method of inductive transfer learning on regression tasks based on differential geometry

Abstract: Transfer learning is a crucial technique for handling a small amount of data that is potentially related to other abundant data. However, most of the existing methods are focused on classification tasks using images and language datasets. Therefore, in order to expand the transfer learning scheme to regression tasks, we propose a novel transfer technique based on differential geometry, namely the Geometrically Aligned Transfer Encoder (${\it GATE}$). In this method, we interpret the latent vectors from the model to exist on a Riemannian curved manifold. We find a proper diffeomorphism between pairs of tasks to ensure that every arbitrary point maps to a locally flat coordinate in the overlapping region, allowing the transfer of knowledge from the source to the target data. This also serves as an effective regularizer for the model to behave in extrapolation regions. In this article, we demonstrate that ${\it GATE}$ outperforms conventional methods and exhibits stable behavior in both the latent space and extrapolation regions for various molecular graph datasets.

Published as a conference paper at ICLR 2024
GEOMETRICALLY ALIGNED TRANSFER ENCODER FOR INDUCTIVE TRANSFER IN REGRESSION TASKS
Sung Moon Ko", Sumin Lee", Dae- WoonWo Jeong', LG Al Research (sungmoon. ko,
Woohyung Lim,
Sehui Han
sumin. lee, dw jeong, w. wimim, hansse (amase.han)al e gresearch.
ABSTRACT
Transfer learning is crucial technique for handling a small amount of data that is potentially related to other abundant data. However, most of the existing methods are focused on classification tasks using images and language datasets. Therefore, in order to expand the transfer learning scheme to regression tasks, we propose a novel transfer technique based on differential geometry, namely the Geometrically Aligned Transfer Encoder (GATE). In this method, we interpret the latent vectors from the model exist on a Riemannian curved manifold. We find proper dif- feomorphism between pairs of tasks to ensure that every arbitrary point maps to locally flat coordinate the overlapping region, allowing the transfer of knowl- edge from the source to the target data. This also serves as an effective regularizer for the model to behave in extrapolation regions. this article, we demonstrate that GATE outperforms conventional methods and exhibits stable behavior in both the latent space and extrapolation regions for various molecular graph datasets.
INTRODUCTION
machine learning model requires abundant data to learn and perform effectively. However, col- lecting sufficient amounts of data often consumes substantial amounts of energy and time. One of the most practical approaches to addressing this issue is to make use of large datasets correlated with the target data. Transfer learning stands out one of the potential contenders. Transfer learning focuses on transferring knowledge from source to target data. Recently, there has been immense development in various domains such as languages (Zhuang et al., 2011 Long et al.; Zhuang al. Pan et al. 2020). computer vision (Quattoni et al. 2008 Kulis et al. [01 Raghu et al, Yu at.
2013
2014
2019
2022) and biomedical domain (Wang al. 2019 Peng et al.) 2021)
Despite these achievements, the primary area of concern is mostly limited to classification tasks in the vision and language domains. Moreover, identifying an architecture that effectively addresses regression problems has proven to be challenging. Hence, our primary objective is to develop an algorithm that can be generally applied to regression problems. In this article, our main focus centers on molecular property prediction tasks (Scarselli et al. 2009; Bruna et al. [0013] Duvenaud et al.)
2015 Defferrard et al. 2016; Jin et al., 2218) Coley et al. 2019 Ko et al. (023). Molecular datasets tend to be small in size, have a large number of task types, and are mainly regression. This makes molecular property prediction good application test our novel algorithm.
The fundamental concept of our method rests on geometric interpretation of the encoding space. Given that the encoding process within a model involves intricate non-linear mappings, is reason- able to regard the encoding space non-linear and curved. As it is illustrated in Figure we assume a data point in each task is on a manifold, and collection formulates a coordinate patch that can be interpreted task coordinate. Each task coordinate has chart to map to another correlated task to form a manifold. Additionally, machine learning models are obligated to exhibit continuity and differentiability to facilitate training through backpropagation. This requirement extends the coding space, which should ideally possess smoothness properties. These observations collectively suggest the possibility of interpreting the encoding space Riemannian manifold.
'These authors contributed equally this work

---

Published as a conference paper at ICLR 2024
Manifold M
Φ1
Φ
Φ1
Φ2
Z1
Z2
Φ1 Φ2
Ø2 +7
Coordinate task,
Coordinate for tasky
Two different coordinate frames are demonstrated in the figure, with coordinate trans- formation maps each individual coordinate choice. One can interpret each coordinate frame as task-specific coordinates and map them with transformation models. An arbitrary point in the over- lapping region of the manifold can transformed from one task coordinate another by combining mapping functions ø. Moreover, by introducing perturbation points, as demonstrated in the figure, one can define the distance between points to match the geometrical shape in the overlapping region.
Figure 1:
In general, curved space is considerably more intricate than a flat space. For instance, when deal- ing with curved space, determining the distance between points requires the computation of entire geodesic equations based on specific metric. Consequently, the straightforward calculation of sim- ilarities between data points using Euclidean distance longer applies, making it challenging to discern meaningful similarities between points. However, when the manifold is regarded as a Rie- mannian manifold, there strategies that can help navigate these challenges. One such strategy involves finding diffeomorphisms that allow points to situated in a locally flat frame (LF). As a result, it becomes feasible to develop models specifically designed identify appropriate diffeo- morphisms. These models aim to confine individual data points to LF within overlapping regions that align with other coordinate patches. This approach simplifies the process of establishing cor- respondence between similar points across different coordinate systems, as each point is effectively situated within a Euclidean environment on a local scale.
In essence, the act of matching overlapping points on a manifold ensures the alignment of their ge- ometries. This alignment crucial because the encoding space's inherent geometry in the source data contains relevant information for the target data. By achieving this alignment, the model fa- cilitates the flow of information from the source to the target. effect, this approach establishes strong foundation for the model's capabilities and extends its capacity to generalize effectively.
Our main contribution of the article is as follows.
We design a novel transfer algorithm (GATE) based on Riemannian differential geometry.
The GATE is utilizable on regression tasks for inductive transfer learning.
The GATE outperforms conventional methods in various molecular property regressions.
The GATE exhibits stable underlying geometry and robust behavior in extrapolation tasks.
2 RELATED WORKS
2.1 TRANSFER LEARNING
Given a source domain D with source task T and a target domain D with target task T transfer learning aims to improve the learning of a better mapping function f*(.) for the target task 7" with the transferable knowledge gained from the D* and T (aan Yang 2009). Depending on the be divided into
relationship between the source and target domains and tasks, transfer learning three categories: inductive, transductive, and unsupervised transfer lcarning. Our problem setting inductive transfer learning, where the target task is different from the source task (7 * T³),
2

---

Published as a conference paper at ICLR 2024
regardless of whether the source and target domains are the same Yang al. 2020 molecular property prediction is a regression task. In transductive transfer learning, studies have been done regression tasks such domain adaptation regression (DAR) (Chen et al. [021 However, recent studies on regression tasks in an inductive transfer learning setting were only found in papers that applied it to specific domain (Li & Fourches 2020 Hoffmann al. 2023)
Additionally,
2.2 MULTI-TASK LEARNING AND KNOWLEDGE DISTILLATION
Multi-task learning (MTL.) can be viewed as a form of inductive transfer learning that learns multiple related tasks simultaneously. By using a shared representation, the knowledge gained from one task can help the learning of other tasks (Caruana) 1997). MTL and transfer learning both aim to generalize knowledge across different tasks. However, MTL aims to improve performance on a set of target tasks with equal importance, whereas transfer learning only focuses
target task. Studies using MTL in various domains can be found (Lee & Kim 2019: Liu et al. (0022).
Knowledge distillation (KD) is training paradigm where the knowledge from a large teacher model is transferred to a small student model. According to (Gou et al. [021], distillation techniques can be classified by knowledge types: espponse-bssede feature-based, and related-based knowledge. sapponse-based knowledge, also called "dark knowledge", uses the output of the last layer, such as logits (Hinton al., 2015). Feature-based knowledge distillation utilizes the output of intermediate layers, which are feature representations, to guide the student model. Gelation-baser KD transfers the relationships between features or data samples. For molecular property prediction, (Joshí al, 2022) introduced feature-based distillation for GNNs by preserving global topology. Studies using cross-modal KD can also be found (Zhang et al. 2022 Zeng et al. 2023).
2.3 GEOMETRICAL DEEP LEARNING
Methods that extend deep neural networks to Euclidean and non-Buclidean domains, such as graphs and manifolds, are collectively referred to as geometric deep learningGDDL) (Bronstein et al. 2017). Regarding "how transfer", transfer learning can be categorized into instance-based, feature- model-based, and relation-basee transfer learning. A Srature-based approach aims to find a feature representation that is effective for both the source and target domains. In the context of domain adaptation, the subspace spanned by the features in the source and target domains is the knowledge that is transferred (Yang et al.) 2020). Manifold feature learning can learn tight representations that are invariant across domains. Some studies have used the property of manifold to perform unsu- pervised domain adaptation. By embedding datasets into Grassmann mainfolds, (Gopalan et al.) 2011) obtained intermediate subspaces by sampling points along the geodesic between the source and target subspaces. (Gong et al. 2012) extended on this by learning a geodesic flow kernel (GFK) between domains. (Baktashmotlagh
al.) 2014) embedded the probability distributions mannian manifold and used a Hellinger distance to approximate the geodesic distance. (Luo et al.) 2020) proposed Riemannain manifold embedding and alignment framework that used intra-class similarity and manifold metric alignment loss to achieve discriminability and transfcrability, respec- tively. However, our best knowledge, we could not find geometrical deep learning approaches for inductive transfer learning in regression tasks.
a Rie-
3 GEOMETRICALLY ALIGNED TRANSFER ENCODER
The latent vector is believed to capture the essence of information for given task. Therefore, one may consider two different but correlated tasks may share a similar underlying geometry of latent spaces. Hence, if the latent spaces of given tasks can be aligned smoothly, mutual information will flow through the latent spaces from one another. This will lead to the superior performance of a target task not only in interpolation but also in extrapolation cases, which is the basic strategy of the GATE. However, deep learning models are designed to mimic non-linear functions by excavating information from the training dataset. The underlying geometry of models should inherently exhibit non-linearity and non-trivial curvature, as well as latent spaces, which are induced by non-linear models. This makes it challenging to match the underlying geometry of latent spaces.
There are several assumptions required to put geometrical concept into practice: 1) A pair of source and target tasks should be correlated; a pair of tasks should have overlapping region; 3)
3

---

Published as a conference paper at ICLR 2024
Embedding
X
perturbation
distance
Encader (Source)
Encoder (Target)
Transfer (Source)
LF
Transfer (Target)
Z
Zt
Transfer- (Source)
Transfer -1 (Target)
(Source)
Head (Target)
41
leot
Teg + alauto + Bloons
+ Mmmap
8lais
y$
yr
Figure 2: Schematic diagram for the GATE algorithm
mathematically well-defined underlying geometry is preferred, such as a Riemannian. Molecular datasets provide perfect proving ground since they satisfy the first two assumptions, and with some proper constraints, the third assumption can also be established. First, there exist numerous sets of correlated tasks proven by scientific research. Also, the majority of molecules possess values for multiple properties, which ensures a large area of overlapping regions across a pair of tasks. Finally, by imposing the mathematical characters of Riemannian geometry as constraints, the underlying geometry can be interpreted as a Riemannian manifold. Moreover, every molecule can be expressed single corresponding universal string in, for instance, the SMILES format Weininger 1988)
3.1 RIEMANNIAN GEOMETRY
In Figure we first take an input SMILES and embed into the corresponding vector. After em- bedding, latent space is formulated by encoders, which consist of DMPNN Yang et al.) (2019) and MLP layers. The latent vector is fed into nos-ccoresponning heads for inference properties. Here we utilize MSE for basic regression loss in the training scheme as follows:
Areg Σ (v 0,)²
Where N, yr. and dt are the number of data points, target, and predicted value, respectively. This is simple, basic loss to train molecular property, yet match the latent space and design a transfer scheme requires more mathematically advanced techniques.
The Riemannian manifold enjoys isometries induced by diffeomorphisms, which guarantee freedom of coordinate choices on the manifold. Thus, one can find a LF around a point on a manifold. A LF means the coordinate patch Note that the input vector can be embedded on a smooth manifold, and its latent space can also be assumed be smooth since the MI. model should always be smooth an arbitrary point due the backpropagation training scheme. Hence, by utilizing the model to map encoder space to LF, the inner product of vectors satisfies a positive semi-definite property. Considering these aspects, can fix the latent space to Riemannian. This whole process is closely described in later sections.
be interpreted as Euclidean around its infinitesimal boundary.
3.2 MATCHING COORDINATE PATCHES
As we mentioned earlier, we interpret the latent vector from the encoder model to be Rieman- nian manifold. Moreover, we assume each downstream task corresponds to a specific coordinate patch on the manifold with a vast overlapping region in molecular property prediction cases. Since one can always find a mapping relation between different coordinate patches, it is crucial to find the specific analytic form of the mapping function. Yet, it is almost impossible to find analytic form of the mapping function of an unknown manifold with little numerical information. Hence, we utilize deep learning model to learn the mapping relation instead of finding with hands.
4

---

Published as a conference paper at ICLR 2024
The coordinate mapping is induced by Jacobian point. Since vector enjoys isometry dif- feomorphism on a Riemannian manifold and is written in fundamental representation, the coordinate transformation can be expressed as follows:
0z Ozi 2d
Σ
(2)
By introducing model that mimics the above Jacobian and predicts the transformation matrix for given vector, one can freely transform between coordinates. However, its inverse may not be stable, and moreover, its derivative may also not be stable and cause instability in the backpropagation process. Resolving the issue is critical for training the model. Therefore, we tweak the problem to not just mimic the Jacobian itself but predict transformed vector. Furthermore, we utilize an autoencoder scheme to ensure its inverse behavior is accurate and stable.
Transfer(2))
Transfer (z')
(3)
We indeed utilize MSE loss for the autoencoder which consists of transfer and its inverse modules.
lauto MSE(z,
(4)
Here, we restrict the model by imposing an autoencoder loss to match the overlapping region of coordinates. But this is insufficient; thus, stronger conditions are necessary to match coordinates.
Since we are expecting transformations occur between coordinate patches, the transfer and inverse modules should have the same input and output dimensions. Moreover, as we are trying to match the underlying geometry of overlapping regions, which are fact the latent space of the autoencoder, can realize that we require another condition for the autoencoder. We consider two distinct autoencoders for source and target coordinates that map coordinate patches to same LF back and forth; the transfer output from both should be equal in order match the geometry of the overlapping region.
2 Tra
2g Transfer
LF (F
(5)
2/ Transfer(-s
2 Transfer -1 LF (2())
(6)
Where Model,-LLF indicate a model from source to LF frame and vice versa.
Loons MSE(= =()
This loss, we call it consistency loss, pivots points on overlapping regions to glue the underlying geometry. Furthermore, one can induce information flow from the source domain the target domain by utilizing mixture of autodencoders.
(Trasser,-sIFF(3))
E, = Transfer -FF-((=
The above equation indicates a vector from the source domain that is transformed to the target domain, retaining its mutual information. Therefore, this mapped vector d should also predict the same label as the original vector 2, on the target side. This fact gives rise to another form of loss function, namely mapping loss.
Mmap MSE(yt (o-+e)
Here Ye indicates the label for corresponding vectors and ÿ--st indicates predicted value from d Above loss ensures mutual information flow by matching overlapping coordinates on pivot points. This particular loss is applied to reverse case too which maps 2, to Dt-
Now the question is, 'Is this enough?'. The answer is 'No!'. The reason is actually not intricate to find. If we start with immensely much data on both the source and target domains, then the answer will be 'Yes'. However, in most cases, we lack data points to glue coordinates together. If points not enough, there are always overfitting issues that cause the geometry to crumple outside the pivot point region. There are solutions this issue. One is gathering more data points and hoping to work smoothly, and the other is by adding a stronger condition to the geometry.
Our idea is not only to focus point data but also its geometries by considering the geodesics of two different points. Here, there are two issues to compute the geodesics in curved space: one is that the space itself is curved, and the Pythagorean theorem longer holds the space in general.
5

---

Published as a conference paper at ICLR 2024
The other is that the shortest path between two points in curved space is no longer intuitive. In curved space, one can calculate distance using the following equation:
5² 5² -² -ー
Guvdx dx"
(10)
Here, Yuu is a metric on the surface, datt, deL are infinitesimal displacements, and / is the geodesic path between two points. The geodesic of curved space can be defined as a curve whose parallel transport along the curve preserves the tangent vector to the curve. Which indicates,
V (t(o)((t) =0
(11)
Where 7(t is a curve on a manifold, i(t) is the derivative respect to a parameter t. and V is covariant derivative with an affine connection T It is clear that solving the geodesic equation is not a simple task since one must know the explicit form of the metric of space, and the equation itself a non-linear partial differential equation. Thus, as depicted Figure we introduce infinitesimal perturbation points around a data point to detour the issue. The infinitesimal perturbation makes things easy to tackle since a data point can always be expressed on LF, where geodesic becomes trivial. Then can easily compute the distance between data point and perturbed points since it can be interpreted to be on a flat Euclidean space.
The distance between two different points space is key quantity to match the task coordinates since, as shown in eq. 10 distance always requires metric on a space. By restricting neighbor points to infinitesimal region around pivot data point, the metric now be considered Euclidean metric, Tue. Than the eq. simpliffeed as follows:
[(" da.
52 5²=
|E> P
dul duy
d===== due dx =
(12)
Where a and are the pivot data point and its perturbation point, respectively. If perturbation is infinitesimal, one can take off the integral and find displacement in a simplified form.
S = lb
(13)
Since it is possible to find a LF on Riemannian manifold, for given data point, two different task coordinates can always be transformed by their diffeomorphisms to be on a flat coordinate. utilize the above distance between pivot and perturbation points, each task coordinate is now glued together by not only given pivot point but also by their neighbors. It is closely depicted in Figure Right after the embedding, we find perturbation points around given data points and feed them to each task-specific model for encoding. After encoding, models will be entangled together by mapping loss, consistency loss, and distance loss, forcing the distances between pivot points and their perturbations to be equal between tasks.
M
Ldss
1
Σ MSE(s st)
(14)
M
Where M is the number of perturbations and sy is the displacement between pivot data points and their perturbations.
I - -=t")
(15)
2
N Transfer, (Encoder,(x")) (Enncfeer__u
(16)
(17)
Here x² denotes ith perturbation embedded v, and Encoder, and Encoder, are encoder parts of source and target model respectively. Finally, by gathering all losses with individual hyperparame- ters, we obtain the complete form of the loss function used in the GATE algorithm.
1et = freg + alauto + Blcons + Ymmap + 8ldie
(18)
Each hyperparameter tunes the corresponding weight of loss. In most cases, is sufficient to let all the hyperparameters to 1, but in some cases, one can fine-tune the hyperparameters to achieve superior performance. The most critical hyperparameter is 5 since the distance loss manages the regularization effect of the model. This result will be closely examined in the ablation section. And with the proper hyperparameters, our novel transfer model GATE outperforms conventional models in inductive transfer settings for regression tasks, shown in the experiment section.

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
To test our algorithms, we used a total of 14 datasets from three different open databases named PubChem(Kim al. 2022). Ochem(Sushke)
al.) (011), and CII 2022), described in Appendix Table 3. The training and test split is an 80:20 ratio, and two sets of data were prepared: random split and scaffold- -asddd Bemiss & Murcko 1996). Every experiment is tested in four- fold cross-validation setting with uniform sampling for accurate evaluation, and single NVIDIA A40 is used for the experiments. For the evaluation, compare the performance of GATE against that of single task learning (STL), MTL, KD, global structure preserving loss based KD (GSP- KD) (Joshi al., 2022), and transfer learning (retrain all or head network only), We used the same architecture for encoders and heads in both the baselines and our model for all experiments. More detailed experimental setups can be found in Appendix. We performed experiments with total of 23 target and source task pairs, respectively.
4.2
RESULTS
OK
05
AP
DK
D5
AP
WW
140
NV
NV
0.5
05
GIGATE
10
MN
200
50
0KD
CSSPKD
Trarsker
POL
20
20
CC3
43
520
Random split
Scaffold split
Figure 3: The RMSE of GATE divided by the RMSE of the corresponding methods are illustrated in the radar chart. Experiments were performed with 23 target and source task pairs in random split (left) and scaffold split (right) datasets. Inner characters correspond to the target tasks, and outer characters are for the source tasks. The full names of the tasks can be found in Appendix Table
The result of the training for the 23 task pairs with both random and scaffold split is demonstrated in 3 The figure illustrates the relative regression accuracy, calculated as the RMSE of GATE divided by the RMSE of the corresponding method. As shown in the figure, in the majority of cases, GATE outperforms baseline methods by significant margins. Note that, even though the KD and GSP-KD outperforms the GATE for some task pairs, they show critically low accuracy for several task pairs On the other hand, as we can see in the Appendix Table 4-7, GATE exhibited the best performance in more than half of task pairs, specifically scaffold split out of 23 pairs. Considering that the method with the second-highed number of best- performing task pairs had only task pairs in random split and task pairs in scaffold split, this can be regarded as a dramatic difference. Also, when considering the second-best performance, GATE maintained consistently high performance in the majority of pairs, reaching 18 pairs in random split and scaffold split. For the average RMSE over all task pairs, GATE demonstrates significantly lower RMSE compared to the recond-bess method, 9.8 % lower in random split (GSP-KD) and 14.3 % lower in scaffold split (MTL). This indicates that GATE performs even better in extrapolation tasks.
pairs in random split and 13 pairs in
7

---

Published as a conference paper at ICLR 2024
Only Mapping
Mapping Consistency
1.0
training
Draving
va8dudon
MSE
MSE
Elese (6iiinn 1655
lowest validiton loss
100
200
300
400
500
100
200
300
400
500
Epochs
Epochs
Mapping Consistency Distance
Multitask Learning
1.0
1.0
traning
0.8
salicatton
0.8
0.6
0.6
MSE
MSE
0.4
0.4
0.2
0.2
ynlidittion loss
validation
GATE
100
200
300
400
500
100
200
300
400
500
Epochs
Epochs
Figure 4: Training and validation loss curves of GATE and MTL models with different sets of loss terms. Upper left: GATE trained without consistency and distance loss; upper right: GATE trained without distance loss; lower left: plain GATE, lower right: plain MTL.
ABLATIONS STUDIES AND FURTHER ANALYSIS
5.1 ROLE DISTANCE LOSS
The most distinguishable aspect of GATE is the distance loss, which connects task coordinates not only through labeled data points but also surrounding perturbation points. Since learning signals from consistency and distance loss do not require regression target labels, these two losses can be applied an unsupervised manner. GATE directly leverages this advantage in distance loss and regularizes the overall latent space to be geometrically transferable trained GATE using different sets of losses: 1) only mapping, 2) mapping and consistency, and 3) mapping, consistency, and distance losses. As result, we observed that the addition of distance loss significantly suppresses overfitting during the training process, as shown in Figure 4 Note that the training loss curve of MTL never reached the lowest validation loss of GATE (grey dashed line). These results indicate that the regression model can be strongly regularized by the distance loss while having minimal adverse impact on the regression task itself.
LF. To verify this,
Surface tension
Kovats retention index
Viscosity
Abraham descriptor S
2.0
1.0
GATE
0.
-10
200
1.0
MTL
00
Figure 5: PCA on latent space of GATE and MTL model trained with collision cross section dataset as target task and surface tension, Kovats retention index, viscosity, and Abraham descriptor S as source tasks, respectively. The color indicates the predicted value of the collision cross section.
8

---

Published as a conference paper at ICLR 2024
5.2 STABILITY OF LATENT SPACE
For further investigation on the regularization effect of GATE, we examined the shape of the la- tent space of the GATE and MTL models. Transfer learning offers way to leverage information from source tasks, but undesired interfering information can cause negative transfer. Ideally. if a model well-guided by the right information and regularized properly, the overall geometry of the latent space may remain stable and not depend on the type of source tasks. (Zhang et al. [223] Gömez-Bombarelli al,, 2018) However, if the target task is overwhelmed by the source task and regularization is not enough, latent space will be deformed according to the source tasks. To check this effect, we trained regression on the same target task using GATE or MTL with four different source tasks and then performed principal component analysis. Figure effectively illustrates that the latent space GATE exhibits relatively stable characteristics compared to that of MTL.
5.3 ROBUSTNESS TO DATA CORRUPTION
We examined the robustness against data comuu-- tion in order to directly verify the regularization effect. To artificially introduce significant cor- ruption to the dataset, data points with labeled values outside the standard deviation domly selected, and the values of these data points were altered be twice the standard deviation with negation. And then, after training with the corrupted dataset, we assessed the mean squared error (MSE) between the model's predictions on the corrupted data points used during training and the original values before the corruption. If the model exhibits overfitting to the corrupted data, will manifest substantial errors, whereas effective model regularization will lead to error reduction. Figure [6) shows the result of the test for GATE and MTL. The GATE model exhibits considerably lower errors compared MTL. This result indi- cates that the regularization effect of the GATE al- gorithm leads to high robustness against data cor- ruption and successful generalization.
Heat vaporization
Collision cross section
ran-
0.55
2.6
0.5
2.4
0.45
2.2
0.4
MTL
GATE
MTL
GATE
Figure 6: GATE and MTL models are trained with artificially corrupted data points and MSEs between predicted values and original values before the corruption is recorded. The target task is heat of vaporization (left) or collision cross-section (right), with the opposite side be- ing used as the source task.
6 DISCUSSION
We demonstrated novel transfer algorithm based on the geometrical interpretation of latent space. By assuming latent space to be a Riemannian manifold with curved metric, the conventional Euclidean interpretation is no longer valid. However, since diffcommrphism invariance guarantees freedom of coordinate choices, one always find LF for each task coordinate. Hence, it is possible to impose consistency loss by matching an encoded vector from one task to another. Fur- thermore, we showed that the distance loss of displacement from a point in an overlapping region to its perturbations is crucial. All together, we demonstrated that the GATE outperforms conventional methods in the majority of molecular prediction task setups and proved its robustness.
In this article, we focused on regression tasks for molecular property prediction. Yet, the algorithm itself is not restricted to regression tasks. Thus, is interesting to expand the algorithm other domains such as images, languages, further, multi-modal setups. Another technical point is that we considered the neighboring region of a pivoting point to compute displacement on flat space. However, to match the entire geometry, it is necessary to consider not only perturbations but also points in finite-distance regions. this scheme, one must solve a geodesic equation instead of merely computing the Euclidean distance. It is incomparably complicated since one must consider 1) finding an explicit form curved metric and 2) solving highly non-lineat differential geodesic equations analytically numerically. Yet, considering GATE's performance, by imposing geometric characters on fully curved backgrounds, the performance gains may be substantial. Hence, this could be another interesting research topic to expand GATE's horizons.

---

Published as a conference paper at ICLR 2024
REFERENCES
Mahsa Baktashmotlagh, Mehrtash T Harandi, Brian C. Lovell, and Mathieu Salzmann. Domain adaptation on the statistical manifold. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2481-2488, 2014.
Guy Bemis and Mark A Murcko. The properties of known drugs. 1. molecular frameworks. Journal of medicinal chemistry, 39(15))8887-2895 1996.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pieme Vandergheynst. Geomet- ric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-44 2017.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann Lecun. Spectral networks and locally connected networks on graphs. 12 2013.
Rich Caruana. Multitask learning. Machine learning, 28:41-75, 1997.
Xinyang Chen, Sinan Wang, Jianmin Wang, and Mingsheng Long. Representation subspace distance for domain adaptation regression. In ICML, pp. 1749 1759, 2021.
Connor W. Coley, Wengong Jin, Luke Rogers, Timothy F. Jamison, Tommi S. Jaakkola, William H. Green, Regina Barzilay, and Klavs F. Jensen. A graph-convolutional neural network model for the prediction of chemical reactivity. Chem. Sci., 10:370-377, 2019. doi: 10. I..00//C8CC0 URL http: doi. 10. 1ttp///00.113///00222220000
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks graphs with fast localized spectral filtering. 06 2016.
David Duvenaud, Dougal Maclaurin, Jorge Sguillra--parraguiiee Rafael Gómez- Bombarell,, Timo- thy Hirzel, Alán Aspuru-Guzik, and Ryan Adams. Convolutional networks on graphs for learning molecular fingerprints. Advances in Neural Information Processing Systems (NIPS), 13, 09 2015.
Rafael Gómez- Bombarell,,,, Jennifer N Wei, David Duvenaud, José Miguel Hernández- Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge eguilera-Iparraguire Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik, Automatic chemical design using data-driven contin- uous representation of molecules. ACS central science, 4(2):268 276, 2018.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kemell for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2066-2073. IEEE, 2012.
Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Domain adaptation for object recognition: An unsupervised approach. In 2011 international conference on computer vision, pp. 1006. IEEE, 2011.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789-1819, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http::/arxxx
org/ arq/abs/ /ac///5033 1503. 02531
Noah Hoffmann, Jonathan Schmidt, Silvana Botti, and Miguel AL Marques. Transfer learning on large datasets for the accurate prediction of material properties. arXiv preprint urXiv:2303.03000 2023.
Russell D. Johnson III. Nist computational chemistry comparison and benchmark database. NIST Standard Reference Database, 101. 2022.
mist. gov/
Wengong Jin, Kevin Yang, Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecular optimization, 12 2018.
10

---

Published as a conference paper at ICLR 2024
Chaitanya K Joshi, Fayao Liu, Xu Xun, Jie Lin, and Chuan Sheng Foo. On representation knowledge distillation for graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Ben- jamin A Shoemaker, Paul A Thiessen, Bo Yu, Leonid Zaslavsky, Jian Zhang, and Evan E Bolton. PubChem 2023 update. Nucleic Acids Research, 51DD1)DD133-DD1330, 10 2022. ISSN 0305- 1048. doi: 10. 1093/nar/gkac955.. URL https: /doi http://oooooogg/////// wtt.:/.....//.1111////////o/////
Sung Moon Ko, Sungjun Cho, Dac- Woong Jeong, Sehui Han, Moontac Lec, and Honglak Lee. Grouping matrix based graph pooling with adaptive number of clusters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8334 8342, 2023.
Brian Kulis, Kate Saenko, and Trevor Darrell. What you saw is not what you get: Domain adaptation using asymmetric kernel transforms. CVPR 20/ I, pp. 1785-1792, 2011. URL htttps//pp/p..
semanti cscho lar. www.or/g/oo.......ooooo.oo wttpg//oopuussii 741 9723
Kyoungyeul Lee and Dongsup Kim. In-silico molecular binding prediction for human drug targets using deep neural multi-task learning. Genes, 10(1 10(11):900 1):906, 2019.
Xinhao Li and Denis Fourches. Inductive transfer learning for molccular activity prediction: Next- gen qsar models with molpmofit. Journal of Cheminformatics, 12(1):1-15, 2020.
Shengchao Liu, Meng Qu, Zuobai Zhang, Huiyu Cai, and Jian Tang. Structured multi-task learn- ing for molecular property prediction. In International conference on artificial intelligence and statistics, pp. 8906-8920. PMLR, 2022.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Wei Cheng, Xiang Zhang, and Wei Wang. Dual Transfer Learning, pp. 551. doi: 10.
URL httpt:ttssssccct //epubs.
s siam. siam.org/ wwtg://o//aooooooooooooooooooooo abs / 11..1...1111111 1137/1 0t1.://.7886119972888
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
You-Wei Luo, Chuan-Xian Ren, Pengfei Ge, Ke-Kun Huang, and Yu-Feng Yu. Unsupervised do- main adaptation via discriminative manifold embedding and alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5029-503245 2020.
Jianhan Pan, Teng Cui, Thue Duy Le, Xiaomei Li, and Jing Zhang. Multi-group transfer learning multiple latent spaces for text classification. IEEE Access, 8:64120-6410,, 2020. doi: 10. 109/ ACCESS. ACCESSS.0000.29844 298.2745571..
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10): 1359, 2009.
Minshi Peng. Yue Li, Brie Wamsley, Yuting Wei, and Kathryn Roeder. Integration and transfer learning of single-cell transcriptomes via cfit. Proceedings of the National Academy of Sciences, 118110)e20223331118, 2021. doi: 10. 10.1077/ppas.2024333 brq/doi/abs/// h0tp///bb11000 1073/ 202
URL https: http::///www.. /ww. pnas.
Ariadna Quattoni, Michael Collins, and Trevor Darrell. Transfer learning for image classification with sparse prototype representations. Proceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Paltern Recognition, 03 2008. doi: 10. 1000/CVP..200.4555577
Maithra Raghu, Chiyuan Zhang, Jon M. Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning with applications to medical imaging. CoRR, abs/1902.07208,, 2019. URL
httt:://arxiw .rg.oogggaaa... 1902.07208 07208
Franco Scarselli, Mareo Gori, Ah Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks a publication of the IEEE Neural Networks Council, 20:61-80, 01 2009. doi: 10. 10.1109/TNN.00 25000..

---

Published as a conference paper at ICLR 2024
Iurii Sushko, Sergii Novotarskyi, Robert Körner, Anil Kumar Pandey, Matthias Rupp, Wolfram Teetz, Stefan Brandmaier, Ahmed Abdelaziz, Volodymyr Prokopenko, Vsevolod Tanchuk, et al. Online chemical modeling environment (ochem): web platform for data storage, model de- velopment and publishing of chemical information. Journal of computer-aided molecular design, 25:533-554 2011.
Jingshu Wang, Divyansh Agarwal, Mo Huang, Gang Hu, Zilu Zhou, Chengzhong Ye, and Nancy Zhang. Data denoising with transfer learning in single-cell transcriptomics. Nature Methods, 16: 875-878, 09 2019. doi: 10.1038/s41 592-019-0537-1
David Weininger. Smiles, chemical language and information system. introduction to method- ology and encoding rules. Journal of chemical information and computer sciences, 28(1):31 1988.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Tim Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi Jaakkola, Klavs Jensen, and Regina Barzilay. Analyzing learned molecular representations for property prediction. Journal of Chemical Information and Modeling. 59, 07 2019. doi: 10. 1a.102/acc..cim..bo00 9h00237..
Qiang Yang, Yu Zhang, Wenyuan Dai, and Sinno Jialin Pan. Transfer learning. Cambridge Univer- sity Press, 2020.
Xiang Yu, Jian Wang, Qing-Qi Hong. Raja Teku, Shui-Hua Wang, and Yu-Dong Zhang. Trans- fer learning for medical images analyses: A survey. Neurocomputing, 489:230-254, 2022. ISSN 0925-2312. doi: http:///oolory//0.00l//jeuuuoemp
159. URL https://
sciencedirect cwm/ htt.a///cc.......aaaaaaa//////2.
Liang Zeng, Lanqing Li, and Jian Li. Molkd: Distilling cross-modal knowledge in chemical reac-
tions for molecular property prediction. arXiv preprinl arXiv:2305.01912 2023,
Hao Zhang, Nan Zhang. Ruixin Zhang, Lei Shen, Yingyi Zhang, and Meng Liu. Coordinating cross-
modal distillation for molecular property prediction. arXiv preprint arXiv:2211.16712,, 2022.
Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu, A survey on negative transfer. JEEF/CAА Journal of Automatica Sinica, 10(2):305-329, 2023. doi: 10.1109//AS..2222100000
Fuzhen Zhuang, Ping Luo. Hui Xiong, Qing He, Yuhong Xiong, and Zhongzhi Shi. Exploiting
associations between word clusters and document classes for cross-domain text categorizationf.
Statistical Analysis and Data Mining: The ASA Data Science Journal, 4(1):100-114, 2011, doi:
https://do..org/II 100000..000.....mmmmmm 10099.
URL https:: https////onnnneebarrrry. wi ley com//oii/
abs abs/10. 1002 sam. 10099
Fuzhen Zhuang, Ping I Luo, Changying Du, Qing He, and Zhongzhi Shi. Triplex transfer learning: Exploiting both shared and distinct concepts for text classification. In Proceedings of the Sixth
ACM International Conference on Web Search and Data Mining, WSDM '13, pp. 434, New York, NY. USA, 2013. Association for Computing Machinery. ISBN 9781450318699, 24334494344
doi: 10.
(1552243339..4333444 /RL/Itts::/6...or///1
Fuzhen Zhuang, Ping Luo, Changying Du, Qing He, Zhongzhi Shi, and Hui Xiong. Triplex transfer learning: Exploiting both shared and distinet concepts for text classification. IEEE Transactions on Cybernetics, 44(7):1 2014. doi: 10.
12

---

Published as a conference paper at ICLR 2024
A NOTATIONS
Our notation is based on index notation and Einstein summation conventions. Notation of functions and matrices in our algorithm follows.
X Vector
X Vector Field
dæ Basis
X # Dual Vector Field
dæ Dual Basis
T : Tensor
Th
m w R4 (p, q) Tensor Field
9p Metric Tensor
8 uv: Kronecker Delta
V Covariant Derivative
C Lie Derivative
1 wv Christoffel Symbol
All indices are raised and lowered by a metric 9Nv- For instances,
g¹ v 9d 9ov
(19)
where
9 9uv 81
(20)
Here D is the number of dimensions.
B PROOFS AND DERIVATIONS
B. THE DEFINITION RIEMANNIAN MANIFOLD
A curved space is complicated to comprehend in general. Since late 19th century, there has been immense development in differential geometry to interpret curved spaces formally. One of the best- known intuitive geometrys is the Riemannian. Riemannian geometry enjoys handful of useful mathematical characters that can be utilized in the real world. The formal definition of Riemannian as follows:
Definition B.1 (Riemannian Manifold). A Riemannian metric smooth manifold M is choice at each point & E M of a positive definite inner product smooth manifold endowed with the metric g a Riemannian manifold, denoted (M,g).
F x T,M + on T.M. The
As it is expressed above, Riemannian manifold is smooth and differentiable everywhere the manifold and its derivative as well. Also, a Riemannian enjoys diffeomorphism invariances, induced by the Lie derivative Lx. One easily notice that the adjoint operation between two different Lie derivatives forms a group, namely the diffeomorphism group. This isometry ensures coordinate choices without changing the global geometry of the space.
OX
X' = XXX X ax dx,, X" =X
(21)
As it is depicted in eq. 21 transformed vector remains unchanged. Moreover, always fix the transformed coordinate in a locally flat space.
den
g"
X
(22)
0x
13

---

Published as a conference paper at ICLR 2024
Where 5d is vector locally flat frame. To ensure the vector is flat frame, one must impose the following condition:
8²
5 =0
(23)
0t
Since vector is flat frame, it should be in free-falling motion, so its acceleration should be trivial. On a locally flat frame, the metric also becomes flat Euclidean metric
9ur p
(24)
B.2 COVARIANCE
The vector should be transformed in the same manner in any coordinate frame. However, if the space no longer flat, the ordinary derivative no longer guarantees it. Let us consider a derivative of vector in a general curved space.
Or 0,
0 + 0 Or
(25)
Where 0,
then the vector transformation can be written as follows: 8x
a
Ox
0x'" VP)
0. + -0''' = 2r Ox Orr VP)
(26)
0x"
dx'
82''
dx- VP)
(27)
dx Ox"
As it is shown above, transformation of vector curved space with ordinary derivative is no longer covariant. Thus, one must impose an additional factor to make it covariant, namely an Affine connection. With this factor, one can define covariant derivative, replacing an ordinary one.
0. + P 10
(28)
By requiring a covariance condition on the covariant derivative,
dz 0x'
V
3r Ox V VoV" V
(29)
Then one can induce the explicit form of connection.
V" 0 V²
+
KA VA
(30)
Under coordinate transformation,
2 +
Or Ox
dx 8x'
I no
V'
dr'H Or 0,V*
Ox'' OxPda V
I
V
(31)
Dz'H
dx
po
Here, to make the derivative of a vector covariant, following equation must hold:
Or Or' Oxp v2
82'
Ox Ox" Ox' Or pa
via + na
r
Va
(32)
Which is
m Or VT)
Ox you
Or Ox J2 Er' dr dxp Vd
Pd
VO
(33)
pa dxr
Fr' dx pr
m V Ox dx dx
Ox Or Ox"
tx dz' dx pa v
dr dr'n Oz VA
(34)
This leads us to the explicit form of how the Christoffel symbol transforms under coordinate changes.
Ox Oxp Ox Ox' Dx's Or. T Ox dx 8r'"
(35)
Dr'N dx'' Ox"Or"
Since the Kronecker delta is constant matrix, it is obvious that the derivative of the delta should be trivial. Then one can apply the chain rule to the delta and find the following relation, which can simplify the above transformation rule.
θ
ar ② Ox Ox Fx Dxnn Ox
2x dx 8x4 dx Ox Pz 0
Dr' 84
Fr' D..
Dx Dr't dx dr dr dr Dz!p (36)
14

---

Published as a conference paper at ICLR 2024
Finally, the transformation rule for Christoffel symbol is follows:
Ox Ox JAN 0r'"
dx
Ox' xx
m
r
(37)
Dr'v' Or pr
dr Ox'Ox''
By the same logic, one can easily find out how covariant derivatives act on forms.
VuV. ONV. - D uM VA
(38)
B.3 EXPLICIT FORM OF CHRISTOFFEL SYMBOL
The metric is ruler of a given geometry; it should not vary under position on coordinate. The Fuelidean trivial to see since the metric on Fuclidean space is mere 6pws which is constant matrix.
0 0x
pp = 0
(39)
However, in the curved case, the above statement should also hold to interpret the metric as a ruler, yet the statement does not hold for an ordinary derivative. There, the covariant derivative kicks in to replace an ordinary derivative instead. By taking covariant derivative to the curved metric, the term diminishes.
Valu 0
(40)
One can express this in terms a flat metric with a diffeomorphism transformation factor.
ae^ dE Ox Ox" (E)
Inw (x)
(41)
If we take a derivative of x on both sides, the above equation becomes:
① Ox Ox (x)
Oe de Ox" Ox Exp + Ox" Ozy 6ro
Ox Eo
(42)
g& Or OED ded
Og Ox de* dge
8xp dr drl de> Ox Or 8xp
(43)
Ox" DEP Ox Dra Sz Ox- dx" DEP 9ur
OREA dx dx" Tex
(44)
From 40 one can easily find out the specific form of the Christoffel symbol in terms of derivatives of curved and flat coordinates.
O Ipv = - apITv + voDua
(45)
O222 Ox
1 oμ Ox Out dEx (x)
(46)
Since the metric should always be symmetrie, the lower indices of the Christoffel symbol should also be symmetric. It is called a torsion-free condition. Furthermore, by utilizing a simple mathematical trick, one obtain the Christoffel symbol in terms of the metric Jae.
F 9uv
Gu + D auSur P
(47)
i
Iva
1T
uuIra +1" polur
(48)
Oru
θ
Fr Iap
Pr
Doorr + pr
(49)
Adding the first two equations and subtracting the last one leads to
r pv = 29 00 Sup
Sup +
dr Sow Iuv)
(50)
15

---

Published as a conference paper at ICLR 2024
B.4 GEODESIC EQUATIONS
The shortest path between two points is simple in flat space. However, in curved space, the notion becomes rather complicated. The shortest path in curved space is defined geodesic. There several ways to induce a geodesic equation. One is by requiring a free-falling condition.
(T)
(51)
θr
By diffeomorphism, one can transform a coordinate into an arbitrary coordinate x.
0 à DE Or" 05 dr Or" Ør Ox d72
Px"
Or Ox"
+
(52)
Or~Oz"
Òr dr Ox
gR Oz Or Ox"
+
(53)
O,2
Or>Ox" dEn Ōr Ør
O,2
Another way to derive the equation is by finding the minimum value of the distance in curved space.
S J Vow dxl dxy
dr guv dr dr
(54)
By varying the above equation and requiring it to be O, one can compute its minimum value, and after tedious calculation, the geodesic equation can obtained.
C BASE GRAPH NEURAL NETWORK MODEL
In general, molecule represented graph form. Therefore, order handle molecule dataset, it is inevitable to utilize graph neural networks. We chose directional message passing network (DMPNN) (Yang et al.) 2019) for our backbone, since it outperforms other GNN architectures in molecular domain. Given graph, DMPNN initializes the hidden state of each edge (i,j) based on its edge feature Es with node feature X,. At each step t, directional edge summarizes incident edges as a message m 1+1 and updates its hidden state to ++1
e+1 m
Σ
(55)
REN
ReLU(h) + W.m
(56)
Where (i) denotes the set of neighboring nodes and We learnable weight.he hidden states of nodes updated by aggregating the hidden states of incident edges into message m and passing its concatenation with the node feature X, into a linear layer followed by ReLU non-linearity
m m+1
Σ h
(57)
JEN(()
h (ieLUUW,,oncct(XX) m ))
(58)
Similarly, W. denotes learnable weight. Assuming DMPNN runs for T timesteps, we use (Xouty Eout) = GNN(A, X, E) to denote the output representation matrices containing hidden states of all nodes and edges, respectively (i.c., Xout, hy and Eout, h?)).
For graph-level prediction, the node representations after the final GNN layer are typically sum- pooled to obtain single graph representation lG Σ; h4, which is then passed to FFN prediction layer.
D ARCHITECTURE AND HYPERPARAMETERS
Detailed steps training GATE is described Algorithm The architecture of our model is composed of five distinct networks and their parameter sizes depicted in Table As illustrated in Figure [ one embedding network is shared across tasks, and encoder, transfer, inverse transfer, and head network exists for each task. The embedding network embedd(.) has the DMPNN architecture
16

---

Published as a conference paper at ICLR 2024
Algorithm GATE
Initialize encoder network fe. transfer network ft. inverse transfer network head network fr. with random parameters 0
2:
3: for epoch i 1.2...... do
4:
for each l. Taskis do for each batch b (x', E dataset D do a² embedd((x) {ā'} t perturb(a')
5:
6:
7:
8:
9:
2º st(a²)
10:
m² +- ft((t)
{2)) t 12(1a')
12:
{m²) + 11(()
13:
14:
Leeg- + MSELoss(y (2())
Lauto MSELoss (m²), 22))
16:
for each 8 Subtasks do 2" f (a(a m² t fi(2*)
18:
19:
20:
{ t f2(fa*))
21:
(m-) + [1
22:
23:
Tmmap +- Lomap + MSFE Loss(y' ( S/ fi(m²)
24:
Leons Tecons MMFFFssS (omss(mmm (m)) .m²)
25:
Ldist f Ldist + MSELoss(m' (m²),, {m²})
end for
27:
Compute Liotal Lreg + alauto + 3Lmap + Llcons 8Ldist Update 0 using Liotal
29:
30:
end for
31:
end for
32:
end for
with depth and converts the input molecule representation æ into a new representation a in common embedding space. We apply perturbation perturb(-) to a for number of perturbations, which is set to 10 in this paper. All of the perturbed representation {ā} along with a are then fed into the encoder network. The encoder network is composed of backbone network and bottleneck network. Backbone network has the DMPNN architecture with depth 2 and the bottleneck network has an autoencoder structure with MLP layers. The output from the encoder fa becomes the input to the transfer network and head network. The output of transfer network fz(z), notated m, used to calculate consistency loss and distance loss. It is also fed into inverse transfer network, be used to calculate autoencoder loss. The output from head network fn o fmm) is used to calculate regression loss and mapping loss. We trained 600 epochs with batch size 512 while using AdamW (Loshchilov & Hutter 2017) for optimization with learning rate 5e-5. The hyperparameters for o, 8, are 1, respectively.
so that the output from inverse transfer network fi(m)
Table 1: Network parameters
network
layer
input, output size
hidden size
dropout
backbone
DMPNN
[134, 100
200
0
bottleneck
MLP layer
100,
50
0
transter
MLP layer
50, 50
100,
0.2
inverse transfer
MI. layer
50, 50
100, 100,1001100 100
0.2
head
MLP layer
50.
25,12
0.2
17

---

Published as a conference paper at ICLR 2024
Table 2: Hyperparameters
learning rate
0.00005
optimizer batch size epoch # of perturbation
AdamW W
512
600
T0
at %,
1, I, 1,
E DETAILED EXPLANATION OF DATASETS AND EXPERIMENTAL SETUPS
E.I DATASETS
Table 3: Detailed information about the datasets.
name
acronym
source
count
mean
std
Abraham Descriptor S
AS
Ochem
1925
1.05
0.68
Boiling Point
BP
Pubchem
7139
198.99
108.88
Collision Cross Section
CCS
Pubchem
4006
205.06
57.84
Critical Temperature
CT
Ochem
242
626.04
120.96
Dielectric Constant
DK
Ochem
1007
0.80
0.41
Density
DS
Pubchem
3079
1.07
0.29
Enthalpy of Fusion
EF
Ochem
2188
1.32
0.32
Ionization Potential
IP
Pubchem
272
10.00
1.63
Kovats Retention Index
KRI
Pubchem
73507
2071.20
719.34
Log P
LP
Pubchem
28268
11.17
9.89
Polarizability
POL
CCCB
241
0.84
0.26
Surface Tension
ST
Pubchem
379
29.01
10.36
Viscosity
VS
Pubchem
294
0.47
0.87
Heat Vaporization
HV
Pubchem
525
43.77
18.08
We used 14 different molecular property datasets from three different open databases, described in Table 3 and below explanations for evaluation the GATE. Before the training process, the data were purified to exclude data with incorrectly specified units. typos, and extreme measurement environments. All datasets were normalized by mean and standard deviation before the training process. We selected 23 pairs of source and target tasks among the 14 datasets, considering the number of data points in each dataset. We also tried to select task pairs with diversity in correlation shown in the Figure 7 for fair and unbiased examination. Hereby, explicitly describe the physical meaning of each dataset.
AS The solute dipolarityypolaaiixibiitt
BP The temperature which this compound changes state from liquid gas at given atmospheric pressure.
CCS The effective area for the interaction between individual ion and the neutral gas through which is traveling.
CT: The temparature when no gas can become liquid no matter how high the pressure is.
DK The ratio of the electric permeability of the material to the electric permeability of free space.
DS The mass of a unit volume of a compound.
EF The change enthalpy resulting from the addition or removal of heat from mole of substance to change its state from solid to liquid.
18

---

Published as a conference paper at ICLR 2024
Absolute Corr. Values Between Task Pairs
1
0.8
0.7
0.6
0.5
0.4
0.3
0.1
Iovi-
Ary
hyreeg
de-lp
pol-
hive-gt
dkt-gf
38
ca cov-kri
hy--bp
at
10 bp
bp-b
Ir of
hivekkr
kri
bp
Figure Pearson correlation between overlapping data points in target dataset and source dataset.
IP : The amount of energy required to remove an electron from an isolated atom or molecule.
KRI: The rate at which compound is processed through gas chromatography column.
LP: Logarithmic form of the ratio of concentrations of compound in mixture of octanol and water at equilibrium.
POL: The tendency of matter, when subjected an electric field, to acquire an electric dipole moment in proportion to that applied field.
ST The property of the surface of liquid that allows it to resist an external force
VS A measure of fluid's resistance to flow.
HV The quantity of heat that must be absorbed if certain quantity of liquid is vaporized at a constant temperature.
E.2 EXPERIMENTAL. SETUPS
For evaluation of the GATE, we compared the performance of six bascline methods, including STL, MTL, KD, global structure preserving loss based KD (GDP-KD), and transfer learning (retrain all or head network only). All of the baselines share the same base architecture, with few different details according to methods. The MTL shares parameters of backbone and bottleneck for given two tasks, and only head networks are separated. In the case of the KD, latent vectors from the bottleneck used labels for the distillation, and the distillation loss ratio set 0. Graph Contrastive Representation Distillation (G-CRD) contains contrastive loss as well as the GSP loss :Joshi et al.) 2022). However, we only adopt GSP loss since the contrastive loss term is not applicable for regres- sion tasks. For the GSP-KD, node features from the last layer of the backbone network are used to calculate pairwise distances, which are the labels of the distillation process. The loss ratio of the distillation process of the GSP is also set to 0. The maximum epoch is set to be 600, and the best models are selected by early stopping.
F EXPERIMENTAL RESULTS
We express explicit test results this section. A total of task pairs from 14 distinct datasets were tested thoroughly with seven different models. Four tables are depicted to show the full experimental results. The best result emphasized by bold and underlined on each individual result, and the second-best result is underlined. The GATE outperforms other conventional methods by noticeable margin. In random split setup, the GATE wins 52.1 7%% out of total tasks, and for up to second, the GATE wins 78. 26% out of total. In scaffold setup, the GATE wins 56.52% and 78.26 26% respectively.
19

---

Published as a conference paper at ICLR 2024
Table 4: Random Split Result (part
Tasks
GATE
STL
MTL
KD
RMSE
STD
RMSE
STD
RMSE
STD
RMSE
STD
hv - ds 0.9221
0.0612
0,9574
0.0519
0.9782
0.0782
1.3726
0,2930
bp 0.4583
0.0193
0.5125
0.0085
0.4370
0.0119
0.5426
0.0335
ds t kri 0,4145
0.0172
0.4154
0.0045
0.4172
0.0102
0.4403
0.0119
<- vs 0.9116
0.0522
0.9574
0.0519
0.9700
0.1052
1.1995
0.1419
hv 0.5471
0.0719
0.5947
0.0357
0.5535
0.0353
0.5878
0.0264
St <- as 0.6689
0.0413
0.9902
0.0729
1.0272
0.0244
1.1601
0.0396
ds dp 0.4046
0.0142
0.4154
0.0045
0.4133
0.0135
0.4378
0,0086
pol f ds 0.3431
0.0475
0.3460
0.0291
0.4367
0.1213
0.3089
0.0270
bp 0.4457
0.0151
0.5947
0.0357
0.4516
0.0366
0.6076
0.0241
dk +- ef 0,4331
0.0140
0.4331
0.0358
0.4498
0.0126
0.3852
0.0238
as f- CCS 0.4648
0.0139
0.5125
0.0085
0.4677
0.0220
0.5364
0.0211
ct - bp 0.1742
0.0034
0.2549
0.12
0.1707
0.0132
0.1690
0.0079
st CCS 0.9546
0.0452
0.9902
0.0729
1.0361
0.0737
1.1731
0.0730
ccs +- kri 0.2476
0.0034
0.2936
0.0110
0.2524
0.0042
0.2622
0.0117
bp 0.7251
0.0581
0.9574
0.0519
0.7550
0.0432
1.1983
0.1815
VS <- CCS 0.5233
0.0323
0.5947
0.0357
0.5792
0.0228
0.6027
0.0127
<- hv 0.7647
0.0622
0.9902
0.0729
0.7179
0.0259
1.1270
0.0184
hy f- 0,9399
0.0896
0.9574
0.0519
1.1118
0.36
1.5114
0,1845
ip + bp 0.5476
0.0642
0.6695
0.0660
0.6067
0.0345
0.5624
0.0273
hy < ef 0.6131
0.0966
0,9574
0.0519
0.8296
0.0999
1.3659
0.2587
hv +- kri 0.5410
0.0732
0.9574
0.0519
0.8631
0.0354
1.3739
0.2487
kri 0.1658
0.0136
0.2549
0.1247
0.1716
0.0090
0.1586
0.0102
ip + dk 0.6510
0.0381
0.6695
0.0660
0.7083
0.0226
0.5508
0.0100
mean
0.5592
0.0271
0.6642
0.0320
0.6263
0.0414
0.7667
0.0908
Count
Ratio
Count
Ratio
Count
Ratio
Count
Ratio
1st
12
52.17%
0.00%
4.35%
2
8.70%
2nd
18
78.26%
4.35%
30.43%
21.74%
20

---

Published as a conference paper at ICLR 2024
Table 5: Random Split Result (part 2)
Tasks
GSP-KD
Transfer Retrain All
Transfer Retrain Head
RMSE
STD
RMSE
STD
RMSE
STD
hv < ds
0.9321
0.0487
1.0428
0.1165
1.1166
0.0024
as bp 0.5315
0.0151
0.4325
0.0104
0.7712
0.0105
ds t kri 0.4147
0.0063
0.4414
0,0154
0.8842
0.0049
hv <- vs 0.9154
0.0130
0.9937
0.0821
1.0091
0.0181
VS hv 0.5619
0.0223
0.5712
0.0232
0.7215
0.0392
st 4 as 0.9938
0.0141
1.1296
0.1302
1.0045
0.0220
ds +- dp 0.4106
0.0077
0.4280
0.0136
0.9111
0,0022
pol f ds 0.2603
0.0270
0.3741
0.0303
0.9060
0.0141
VS +- bp 0.5932
0.0097
0.5445
0.0239
0.7220
0.0645
dk < ef 0.4230
0.0133
0.3936
0.0164
0,9380
0.0026
+ ссS 0.5457
0.0150
0.4741
0.0148
0.9935
0.0033
c bp 0.2018
0.0093
0.1563
0.0044
0.6847
0.0186
f CCS 0.9595
0.0405
1.1334
0.0687
1.1039
0.0046
Ccs +- kri 0.2698
0.0095
0.2273
0,0016
0.6166
0.0567
hv + bp 0.9051
0.0571
0.8267
0.0417
0.8829
0.0499
<- CCS 0.5269
0.0167
0.4868
0.0119
0.8684
0.0116
st <- hv 0.9618
0.0086
1.0290
0.0945
1.0102
0.0138
hv +- ct 0.9207
0.0112
1.2072
0.0460
1.0302
0,0186
ip - bp
0.4631
0.0037
0.9816
0.2334
0.8732
0.0293
hv < ef
0.8112
0.0463
1.0818
0.1021
0.9616
0.0478
hv +- kri 0.9191
0.0676
0.9080
0.0510
1.0715
0.0145
kri 0.2080
0.0057
0. 0.1661
0.0075
0.8349
0.0279
ip dk 0.5257
0.0192
0.6099
0.0273
1.0336
0.0085
mean
0.6198
0.0212
0.6800
0.0540
0.9108
0.0181
Count
Ratio
Count
Ratio
Count
Ratio
1st
17.39%
4
17.39%
0
0.00%
2nd
10
43.48%
5
21.74%
0
0.00%
21

---

Published as a conference paper at ICLR 2024
Table 6: Scaffold Split Result (part 1)
Tasks
GATE
STL
MTL
RMSE
STD
RMSE
STD
RMSE
STD
RMSE
STD
hv +- ds 0.6939
0.0996
0.6744
0.1079
0.6465
0,0776
0.5920
0.0466
as bp 1.0495
0.0256
1.2828
0.0724
1.1677
0. 0.1068
1.3580
0.0136
ds + kri 0.4395
0.0108
0.4477
0.0052
0.4849
0.0061
0.5409
0.0480
hv + vs 0.7174
0.0796
0.6744
0.1079
0.9954
0.2059
0.8948
0.2294
VS <- hv 0.6120
0.0639
0.9816
0.1267
0.8535
0.0558
1.2597
0.3638
st +- as 0.7540
0.0660
0.8041
0.1062
1.0254
0.0251
1.7083
0. 1618
ds +- Jp 0.4049
0.0102
0.4477
0.0052
0.4517
0.0184
0.5221
0.0328
pol + ds
0.9040
0.0852
0.9604
0.1056
1.4198
0.0796
1.3309
0.1998
VS <- hp 0.6121
0.0297
0.9816
0.1267
0.5686
0.0276
0.9371
0.2386
dk + ef 0.7122
0.0545
0.7028
0.0391
0.6549
0.0210
0.8189
0.0462
CCS ct bp 0.3883
1.1313
0.0496
1.2828
0.0724
1.1197
0.0558
1.3773
0,0781
0.0203
1. 1,4436
0.1150
0.4359
0.0126
1.2459
0.1199
< OCS 0.7281
0.0586
0.8041
0.1062
0.9905
0.0737
1.5402
0.1888
CCS + kri 0.5292
0.0094
0.5489
0.0107
0.5297
0.0083
0.5534
0.0190
hv < bp 0.4821
0.0132
0.6744
0.1079
0.4668
0.0169
0.6271
0.0868
<- CCS 0.6126
0.0671
0.9816
0.1267
0.81 86
0.0790
1.3034
0.5354
st
0.7209
0.0412
0.8041
0.1062
0.7237
0.0276
1.5256
1909
hy ct 0.6579
0.0678
0.6744
0.1079
0,6633
0.0660
0.7925
0.2694
ip + bp 0.4668
0.0179
0.5780
0.1475
0.5540
0.0587
0.4205
0.0240
hv ef 0.6406
0.0335
0.6744
0.1079
0.7879
0.0643
0.6773
0.1553
hv < kri 0.5084
0.0264
0.6744
0.1079
0.6204
0.0269
0.6710
0.1524
kri 0.3902
0.0140
1.4436
0.1150
0.5173
0.0927
1.3392
O. 1076
ip + dk 0.4335
1.0119
0.5780
0.1475
0.5335
0. 0016
0.4975
0.0769
mean
0.6343
0.0270
0.8313
0.0408
0.7404
0.0441
0.9797
0.1218
Count
Ratio
Count
Ratio
Count
Ratio
Count
Ratio
1st
13
56.52%
0
0.00%
13.04%
2
8.70%
2nd
78.26%
3
13.04%
39.13%
2
8.70%
22

---

Published as a conference paper at ICLR 2024
Table 7: Scaffold Split Result (part 2)
Tasks
GSP-KD
Transfer Retrain All
Transfer Retrain Head
RMSE
STD
RMSE
STD
RMSE
STD
hv + ds
0.7606
0,0810
0,8659
0.0788
0.9584
0.0339
as bp 1.2340
0.0294
1.1478
0.0264
1.0935
0.0079
ds + kri 0.4467
0.0104
0.8753
0.1134
1.0928
0.0482
hv + vs 0.6536
0.0345
0.7520
0.1666
0.7924
0.0595
VS <- hv 0.6377
0.0253
0.9217
0.1575
0.9179
0.0539
st +- as 0.9335
0.0954
1.2604
0.0946
1.0780
0.0613
ds +- Jp 0,4685
0.0111
0.4664
0.0121
1.0410
0.0026
pol + ds V8 <- hp 0.6599
0.8475
0.0627
L.0385
0.2146
1.3204
0.0491
0.0204
1.1532
0.1766
1.0135
0.0820
dk + ef 0.6353
0.0171
0.7417
0.0384
0.7963
0.0071
ccs CCS ct bp 1.1837
1.1272
0.0778
1.2925
0.0606
1.4530
0.0143
0.0586
0.5644
0.053
0,9347
0.0316
< ocs 0.7344
0.0187
0.9075
0.0431
1.2596
0.0287
CCS + kri 0.5356
0.0115
0.5640
0.0137
0.7904
0.0159
hv < bp 0.7403
0.0889
0.6093
0.0422
0.8111
0.0251
st CCS 0.8027
0.0159
0.7271
0.0828
1.2282
0.0243
0.7417
0.0206
1.4243
0.0627
1.0047
0.0813
hy ct 0.6428
0.008
0.9499
0.2579
0.8089
0.0532
ip + bp 0.4579
0.0207
0.4419
0.0371
0.9704
0.0399
hv ef 0,5862
0.0375
1.0003
0.1719
0.9503
0.0307
hv <- kri 0,5509
0.0252
0.6560
0.0408
0.9998
0.0311
kri 1.2358
0.0373
1.1124
0.1265
1.2769
0.0193
ip + dk 0.4376
0.0255
0.5248
0.0471
1.0165
0.0521
mean
0.7415
0.0363
0.8694
0.0671
1.0265
0.0217
Count
Ratio
Count
Ratio
Count
Ratio
1st
21.74%
0
0.00%
0
0.00%
2nd
47.83%
2
8.70%
4.35%
23


Reviews:
[{"review_id": "vxeHv6ZTCz", "reviewer": "Reviewer_6Cr1", "summary": "The authors describe a novel formulation of regression transfer learning by embedding the latent space in a Riemannian manifold, allowing the notion of consistency across tasks and the mapping of different points together.", "questions": "- In Section 5.3, it seems that using the word \"significant\" when comparing the GATE and MTL results is not precise enough. Can the authors add a statistical test? \n- In the same section, I would also be interested in re-running this experiment with higher deviations, both of the same sign as the original data, just made more extreme (a data point that is 2 sigma away from the mean is made to be 10 sigma away) and the opposite sign (2 sigma more than the mean is changed to 10 sigma less than the mean). Is there a point at which MTL and GATE have more similar performance, or does the gap increase?\n- How is this approach connected to contrastive learning and metric learning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Novel regularization procedure which also has the potential of being used outside of the scope of this paper.\n- Superior performance when compared to other methods for transfer learning.\n- Intuitive idea and easy to implement.\n- Good experimental section, with a nice exploration of overfitting.", "weaknesses": "- The writing quality needs to be improved, there are both distracting grammar issues and, more importantly, the mathematical formulation of the method and description of the prerequisites for understanding this work have not been adequately presented.\n- Section 5.2 is not well-supported, specifically the assertion \"Ideally, if a model is well-guided by the right information and regularized properly, the overall geometry of the latent space may remain stable and not depend on the type of source tasks. However, if the target task is overwhelmed by the source task and regularization is not enough, latent space will be heavily deformed according to the source tasks\" which needs more detail, or a few corroborating citations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmDhDPBRwi", "reviewer": "Reviewer_Gg1C", "summary": "This paper proposes an approach to improve multi-task learning by training separate learners for each task but transferring latent representations between them under the assumption that the latent spaces can be modeled as Riemannian manifolds.  This is accomplished by introducing encoders and decoders between the latent spaces of the source and target task, and training by use of the original loss function plus a cycle consistency loss for the encoder-decoder pair for each direction, and a distance loss based on perturbations to the input to ensure that the latent space mapping from source to target preserves the local metric.  The resulting method, GATE, is applied to a set of molecular prediction tasks, and compared to existing methods such as single-task learning, multi-task learning, transfer learning, knowledge distillation, and global structure-preserving knowledge distillation.", "questions": "1. What surprising results or insights depend on deep findings from differential geometry?\n2. To what degree does GATE succeed at learning a cycle-consistent and metric-preserving map?\n3. What are limitations of the method?  Under what conditions would it do worse than conventional multi-task learning?\n4. What is X'' in equation (7)?\n5. What are the details for how STL, MTL, transfer learning, KD and GSP-KD were trained?\n6. Could the fact that distance loss helps prevent overfitting be related to training with adversarial examples?\n7. What happens if you only remove the cycle-consistency loss, but not the distance loss?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper proposes an interesting approach to transfer learning, and the contribution of each proposed feature (consistency loss, distance loss) is analyzed in an ablation study.  It is very interesting that the distance loss can prevent overfitting.  The choice of a dataset with 14 different tasks is suitable for a multi-task setup.  The use of two different random splits shows careful consideration of the testing setup.  The graphical check of the latent space across tasks helps builds confidence in the method.  The results are reported in detail in the appendix which allows deeper analysis by interested readers.", "weaknesses": "The idea of enforcing cycle-consistency is not new, and it seems appropriate to cite related literature on cycle-consistency within transfer learning such as CyCADA (Hoffman et al. 2018).\n\nI found it difficult to follow the notation introduced to explain the method given the lack of explanation on the notations.  More details are needed to describe the method precisely.\n\nIt is not clear to me how distance loss helps prevent overfitting.  A toy example would help with the intuition.\n\nThe fact that only chemistry applications were considered limits the generality of this paper to folks working in other domains.\n\nIt is not clear how much the concepts of Riemannianian geometry actually add to the paper (what surprising results or insights depend on deep findings from differential geometry?) but instead seem to obfuscate the relatively simple and intuitive ideas which are implemented in the method GATE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tg2RVsDelx", "reviewer": "Reviewer_nPy4", "summary": "This paper tackles the problem of inductive transfer learning in regression tasks. Assuming that the latent vectors of the model lie on a smooth Riemannian manifold, the paper suggests that for effective transfer learning, source and target tasks need to be mapped to regions with large overlap. The paper describes how diffeomorphisms between pairs of tasks can be learned using parametric encoders and decoders, such that individual data points are confined to a locally flat frame in the overlap region. This is a local Euclidean approximation that helps simplify distance calculation when only small perturbations are concerned.\n\nThe proposed method (GATE) adds 3 losses to the regression loss: an auto-encoder loss for the choice of diffeomorphism modeling, a consistency loss that pivots points on the overlapping region, a mapping loss that enforces predictions to be preserved through the transforms, and a distance loss that forces distances between pivot points and perturbations to be equal across tasks. The distance loss can be viewed as a regularizer.\n\nThe proposed method is evaluated on a transfer learning task for molecular property prediction and is shown to outperform alternatives in a majority of the transfer tasks considered. The overall RMSE is also significantly lower than the baseline methods.", "questions": "Notes and questions:\n- The typo in equation 7 was particularly hard to resolve while reviewing this paper. The choice of notation might have contributed to this.\n- It’s not clear what you mean by \"stable characteristics\" in section 5.2. Please elaborate on what makes MTL relatively unstable in this case.\n- Lower bounds are not shown in Figure 6.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality and significance: The Riemannian view of the latent space is likely not new to this work, but two of the loss functions are novel to this work (to the best of my knowledge). The empirical improvement over the baselines in the studied task of molecular property prediction is significant.\n\nQuality and clarity: The paper is overall well motivated. The descriptions are accompanied by formulas and helpful schematic diagrams. The empirical analysis covers several aspects of the proposed solution, including ablation studies to gauge the effect of each part of the loss function.", "weaknesses": "Even though the method is likely applicable and useful in other domains, the paper only studies it on the molecular property prediction task. This significantly cuts into the impact of the paper as the results cannot (in good faith) be extrapolated to a completely different domain of tasks.\n\nWith a single model architecture for molecular property prediction as the only domain, a more detailed description is missing from the main body of the paper (e.g. input/output/latent dimensions, range of values, SMILES format, DMPNN layers, etc.). The paper is not self-contained when it comes to model components.\n\nThe paper is hard to follow in parts, as the overall picture takes more than 4 pages to be completely laid out. I think an overview of all the losses can be included earlier in the paper to help with the flow and clarity of the paper.\n\nSince the heavy machinery of Riemannian geometry is not really used in the paper (apart from freedom for choice of local coordinates and the overall smoothness assumptions), IMO the elaborate dive into the nuances of dealing with geodesics in an arbitrary metric space is unnecessary, if not needlessly confusing. There is no point introducing the Christoffel symbol just to scare off a reader that might think a nice closed-form solution is possible in the generic case. I’d suggest removing derivations and definitions that do not contribute to the flow of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w36UlVPQZQ", "reviewer": "Reviewer_NCqT", "summary": "- The paper introduces a novel transfer algorithm called GATE based on Riemannian differential geometry.\n- GATE is designed for regression tasks in inductive transfer learning and outperforms conventional methods in molecular property regressions.\n- The authors propose a method to match coordinate patches on a manifold using an autoencoder scheme and consistency loss.\n- The approach allows for the flow of information from the source domain to the target domain, improving the model's generalization capabilities.\n\n----\nI would like to thank the authors for the detailed rebuttal.\nAfter reading the rebuttal and other reviewers point, I would like to increase the original score.", "questions": "- The distance loss is simplified as equation 12, which is basically Euclidean distance. This is contradict to the manifold setup. Can the author discussed more about the distance loss, which seems to be the major difference the author proposed compared with other existing methods.\n- The ablation study shows that the problem is an overfitting problem. So it basically means the data is not enough/ early stopping should help. So the real contribution of such method is slightly not distinguishable. \n- Some tiny piece: \n  - 80-20 split of train and test, is not 4-fold cross-validation. \n  - Figure 3 shows GATE as circle, does this mean all equal or it takes GATE as reference?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Experiment demonstrates superior performance of GATE in extrapolation tasks, with 14.3% lower error in scaffold split compared to conventional methods.\n- This paper shows stable underlying geometry of GATE's latent space and robust behavior in the presence of data corruption.\n- This paper provides ablation studies and further analysis to understand the role of distance loss and the stability of the latent space.", "weaknesses": "- The experiments are based on 14 molecular datasets and most counts are limited (largest one contains 73k count, while smallest only has 241). This limits the generalization of such method on other large dataset.\n- Lack of analysis on the computational efficiency or scalability of the GATE algorithm.\n- The proposed method, in abstract, is an encoder-decoder-based method, which is kind of trivial in the transfer learning setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)