SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: Performance comparison of GIN on target molecule datasets, with  $\uparrow$  indicating higher is better and  $\downarrow$  indicating lower is better.
D.2 Takeaway
Grouping more high-affinity datasets improves performance. Previous studies on task grouping have assumed that low-order relationships can be an effective indicator of high-order ones. It suggests that combining multiple source datasets, which can benefit the target dataset, leads to better performance when learned together. Our experimental results also confirm this phenomenon. We take eight datasets as examples and add them one by one with the top three datasets having the highest affinity score as measured by MolGroup. Results are presented in Table 8, where  $\mathrm{Top}\{a,b,\dots \}$  denotes the combination of auxiliary datasets with top-performing ones. We can find that combining more auxiliary datasets leads to better performance in most cases. Besides, training with high-affinity datasets can significantly reduce the variant of FreeSolv  $(1.579\to 0.279)$ , indicating a more robust representation learned from the auxiliary datasets.
Method	SIDER(↑)	ToxCast(↑)	ESOL(↓)	Lipo(↓)	qm9(↓)
Only-target	62.050.021	66.160.004	1.0540.053	0.74320.032	0.02730.001
Beam search(P)	60.800.005	67.360.006	0.9890.058	0.76400.031	0.03990.002
Beam search(P+S)	62.670.009	66.000.003	1.0260.024	0.75240.017	0.02620.000
TAG	63.570.004	65.400.005	1.0150.045	0.75070.014	0.04320.002
Task2vec	60.690.019	63.280.005	0.9970.028	0.75620.015	0.04290.001
MTDNN	61.430.024	65.050.022	1.0450.003	0.77160.032	0.04200.000
UW	58.240.014	62.710.020	1.1200.084	0.78870.067	0.05080.000
Gradnorm	51.650.018	52.400.007	1.4000.096	1.04820.029	0.21540.125
MolGroup	63.750.018	68.680.003	0.9780.047	0.73040.018	0.02730.001
Table 7: Performance comparison using Graphormer on target molecule datasets.
Combinations	ClinTox	Tox21	FreeSolv	BBBP	BACE	ToxCast	ESOL	Lipo
Only Target	56.450.023	74.230.005	3.8421.579	66.620.028	75.020.026	60.690.010	1.5630.040	0.80630.015
+ Top{1}	57.770.028	74.810.006	3.5630.989	67.360.023	71.270.045	62.660.002	1.5020.043	0.80960.014
+ Top{1,2}	57.480.032	75.220.003	3.4620.970	68.640.012	71.130.019	63.180.006	1.5240.077	0.80780.011
+Top{1,2,3}	59.770.027	75.660.004	3.1160.279	68.360.016	77.330.015	63.910.005	1.4020.010	0.79960.005
MolGroup is the hard version of routing mechanism. Here we investigate the use of the routing mechanism in the training of the final model. Specifically, we train the encoder with the routing mechanism on all the auxiliary datasets and adjust the influence between datasets instead of iterative filtering, as shown in Table. 9. It can be observed that the model trained without the negative auxiliary datasets (i.e., MolGroup) outperforms significantly the model trained with negative datasets (i.e., Final model with  $g(\cdot)$ ), although when it is equipped with a routing mechanism. Besides, MolGroup can be considered a hard version of the final model including the routing mechanism. Rather than incrementally eliminating negative training signals, it directly filters out negative auxiliary datasets.
Table 8: Performance of top-k combinations.
BBBP(↑)	ToxCast(↑)	Tox21(↑)	ESOL(↑)	FreeSolv(↓)
Only-target	66.620.028	60.690.010	74.230.005	1.5630.040	3.8421.579
MolGroup	68.360.016	63.910.005	75.660.004	1.4020.037	3.1160.2790
Final Model with g(·)	66.830.030	57.060.011	69.370.018	3.0400.1383	5.4831.3547
Table 9: Comparison with final model trained on all auxiliary datasets.
D.3 Case Study
To give an intuition of the learning process of MolGroup, we show the learning curves of the affinity scores in different filtering rounds in Fig. 8. In each round, auxiliary datasets with affinity scores below 0.6 are removed, and if none of the datasets fall below this threshold, the dataset with the lowest affinity score is filtered out. It can be observed that a significant number of auxiliary datasets are removed in the first or second round. Furthermore, the learning curves tend to converge after 6 epochs in most cases. The dataset PCBA remains in the final round in most cases, indicating its general benefit to the other target datasets. We also notice that the majority of auxiliary datasets are assigned high-affinity scores for FreeSolv, as demonstrated in Fig.1, suggesting that all auxiliary datasets contribute positively to its performance.
Additionally, we plot the learning curves for qm8 and qm9, which don't have suggested auxiliary datasets. As shown in Fig. 9, all the auxiliary datasets are assigned affinity scores lower than the threshold for both qm8 and qm9, resulting in the removal of all datasets after the first round. We attribute this to the large discrepancy in the structure and task between quantum chemistry and the other domains such as medication. Molecules in quantum chemistry have diverse structures, including both natural and hypothetical compounds. They are studied to explore the behaviors of various functional groups. On the other hand, molecules in medication are more focused on structure. Their study revolves around predicting and optimizing molecular properties relevant to drug design.
Figure 8: Learning curves of affinity scores where red dashed line represents threshold.
Figure 9: Learning curves for qm8 and qm9.
E Parameter Analysis
E.1 Analysis on balance coefficient  $\lambda$
To analyze the impact of the balance coefficient  $\lambda$ , we vary  $\lambda$  in the range of  $\{0.9, 0.7, 0.5, 0.3\}$  and pick BBBP, Tox21, and Lipo as examples. The learning curves of the affinity scores corresponding to different values of  $\lambda$  are plotted in Fig.10. From the results, we observe that decreasing  $\lambda$  led to lower affinity scores for the auxiliary datasets, which fails to effectively discriminate the affinity of individual auxiliary datasets. One reason for this is the instability introduced by parameter initialization and the per-step level computation of structure affinity scores. As a result, lower values of  $\lambda$  cause the MolGroup to assign lower affinity scores to stabilize the training of the target dataset. In light of this, we suggest setting  $\lambda$  to a high value, i.e., 0.9.
E.2 Analysis on number of filtering rounds  $R$
As shown in Fig.8, the auxiliary datasets with the highest affinity scores change as we progress through the different filtering rounds. To investigate the impact of the filtering rounds, we test the performance of the top-3 auxiliary datasets selected in each rounds. The comparison results are shown in Table 10 and the selected datasets are illustrated in Fig. 11. We observe that the auxiliary datasets selected in the 3rd round exhibit the best performance. This is attributed to the filtering
Figure 10: Learning curves of affinity scores with different  $\lambda$ .
process in the 1st and 2nd rounds, which removes negative datasets and helps alleviate interference. As a result, MolGroup can estimate the affinity of each auxiliary dataset more accurately, leading to improved performance on the target dataset.
BBBP	Tox21	ClinTox	Lipo	FreeSolv	ToxCast
R=1	67.940.018	75.660.004	57.660.027	0.80130.013	3.22880.543	58.820.007
R=2	67.990.018	75.660.004	57.660.027	0.80200.031	3.22800.627	63.470.003
R=3	68.360.016	75.660.004	59.770.027	0.79960.007	3.1160.279	63.910.005
Table 10: Performance with different number of filtering rounds  $R$  .
Figure 11: Auxiliary datasets with top-3 affinity scores with different  $R$  where we highlight the different edges in these two rounds.
F Broader impact
Impact on machine learning research. We propose a novel strategy that combines the routing mechanism with the meta gradient to quantify the impact of one dataset on another. Previously, the routing mechanism was used to increase the model capacity. Our proposed framework can inspire various extensive applications in machine learning, including neural architecture searching (NAS) and data-centric AI. Specifically, our framework enables the network to determine the optimal routing path through the meta gradient. This empowers the network to control and modify the layerwise architecture, leading to improved performance. Besides, it has the potential to enhance data-centric AI approaches by providing a tool to analyze and understand the relationship between different datasets or sub-datasets.
Impact on biological research. We investigate the relationship between molecule datasets, and, focusing on both the structural and task dimensions, our findings shed light on how different molecule datasets impact each other. The analytical approach we employ can be extended to other biological data, such as protein and RNA. Furthermore, our method has the potential to be utilized for data instance filtering in the biological domain. This is particularly important given that biological data often contains various types of noise. By filtering out data instances with negative effects on downstream tasks, we can improve the quality and reliability of biological data used in research.
Impact on the society. Our study has significant implications for society, particularly in the field of biomedicine and drug discovery. By understanding the relationship between molecule datasets and their impact on each other, we can gain insights into how different molecules interact and influence biological processes. It can aid in the development of more effective drugs and therapies by identifying molecules that have a positive impact on specific biological targets or diseases.