Cross-Quality Few-Shot Transfer for Alloy Yield Strength Prediction: A New Material Science Benchmark and An Integrated Optimization Framework
Authors: Xuxi Chen
Tianlong Chen
Everardo Yeriel Olivares
Kate Elder
Scott K. McCall
Aurelien Pierre Philippe Perron
Joseph T. McKeown
Bhavya Kailkhura
Zhangyang Wang
Brian Gallagher
Venue: Submitted to ICLR 2023 (2023)

Abstract: Discovering high-entropy alloys (HEAs) with high yield strength is an important yet challenging task in material science. However, the yield strength can only be accurately measured by very expensive and time-consuming real-world experiments, hence cannot be acquired at scale. Learning-based methods could facilitate the discovery process, but the lack of a comprehensive dataset on HEA yield strength has created barriers. We present X-Yield, a large-scale material science benchmark with 240 experimentally measured ("high-quality") and over 100K simulated (imperfect or "low-quality") HEA yield strength annotations. Due to the scarcity of experimental annotations and the quality gap in imperfectly simulated data, existing transfer learning methods cannot generalize well on our dataset. We address this cross-quality few-shot transfer problem by leveraging model sparsification "twice" --- as a noise-robust feature learning regularizer at the pre-training stage, and as a data-efficient learning regularizer at the few-shot transfer stage. While the workflow already performs decently with ad-hoc sparsity patterns tuned independently for either stage, we take a step further by proposing a bi-level optimization framework termed Bi-RPT, that jointly learns optimal masks and automatically allocates sparsity levels for both stages. The optimization problem is solved efficiently using gradient unrolling, which is seamlessly integrated with the training process. The effectiveness of Bi-RPT is validated through extensive experiments on our new challenging X-Yield dataset, alongside other synthesized testbeds. Specifically, we achieve an 8.9~19.8% reduction in terms of the test mean squared error and 0.98~1.53% in terms of test accuracy, merely using 5-10% of the experimental data. Codes and sample data are in the supplement.

Under review as a conference paper at ICLR 2023
CROSS-QUALITY FEW-SHOT TRANSFER FOR ALLOY YIELD STRENGTH PREDICTION: A NEW MATERIAL SCIENCE BENCHMARK AND AN INTEGRATED OPTI- MIZATION FRAMEWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Discovering high-entropy alloys (HEAs) with high yield strength is an important yet challenging task in material science. However, the yield strength only be accurately measured by very expensive and time-consuming real-world exper- iments, hence cannot be acquired scale. Learning-baas tate the discovery process, but the lack of a comprehensive dataset on HEA yield strength has created barriers. We present X-Yield, large-scale material science benchmark with 240 experimentally measured ("high-quality") and over 100K simulated (imperfect or "low-qualityy" HEA yield strength annotations. Due to the scarcity of experimental annotations and the quality gap in imperfectly simulated data, existing transfer learning methods cannot generalize well on our dataset. We address this cross-quality few-shot transfer problem by leveraging model sparsifi- cation "twice" - as a noise-robust feature learning regularizer at the pre-training stage, and a data-efficient learning regularizer at the few-shot transfer stage. While the workflow already performs decently with ad-hoc sparsity patterns tuned independently for either stage, take a step further by proposing bi-level op- timization framework termed Bi-RPT, that jointly learns optimal masks and au- tomatically allocates sparsity levels for both stages. The optimization problem is solved efficiently using gradient unrolling, which is seamlessly integrated with the training process. The effectiveness of Bi-RPT is validated through extensive experiments on our new challenging X- Yield dataset, alongside other synthesized testbeds. Specifically, we achieve an 8.9 19.8% reduction terms of the test mean squared error and 0.98 1.53% in terms of test accuracy, merely using 5- 10% of the experimental data. Codes and sample data are the supplement.
methods could facili-
INTRODUCTION
Machine learning (ML. methods have recently demonstrated great promise in the important field of material science, and in this paper, we focus on ML-assisted high-entropy alloy (HEA) (Ych al.) 2004) discovery and property prediction. HEAs own promising properties that traditional alloys do not hold, such as extraordinary mechanical performance at high temperatures, making them well- suited options for various material applications. One particular property, i.e., the yield strength of HEAs, characterizes the maximum stress material can endure before starting to deform, which is critical parameter for customized HEA design.
However, in order to accurately measure the yield strength of specific HEAs, expensive scientific experiments need to be conducted for each alloy, often involving hard-to-create experimental condi- tions, especially at high temperatures (mainly caused by difficulties with oxidation control) as well as extremely long experimental duration. At high temperatures, these measurements are typically taken with the Gleeble system (Gle). From sample preparation to yield strength measurement can take between two to four weeks machining the sample, and preparing and mechanically testing with the Gleeble. Therefore, it is challenging acquire yield strength measurements from those "high-quality" experiments at scale.
for domain expert team, including melting of the alloy,

---

Under review as a conference paper at ICLR 2023
Similar the trends in computer vision fields (Tremblay al. 2018 recent efforts attempt to mitigate the scarcity of real-world measurements using ML-based predictors: to directly predict their yield strengths from the alloy inputs (Bhandari et al., 2021a); and such ML-based predictors could be trained using simulated data. Indeed, material sciences applications are often blessed by developed simulation models, e-g.. Maresca & Curtin (2020). However, such a blessing is often compromised by the domain gap between the simulated data and the "ground-truth experimental data, often due many inevitable simplifications in simulation modeling. For example, the yield strength of material can vary greatly based on processing and testing conditions as well as grain size and texture (Toda-Caude Coda-Carballo et al., 2014 Lin et al. 2014); yet simulation models commonly rely on properties intrinsic to the alloy and do not incorporate variations in experimental conditions. The lack of public datasets in this field also renders difficult to benchmark MI models' progress.
In this paper, we start by curating large-scale benchmark, called X-Yield, that for the first time combines experimental data with simulation data address the problem of predicting yield strength in HEAs. While using experimental data is always preferred since they are "high-quality" ground truths, is impractical to generate high quantities of data, especially for capturing yield strength at elevated temperatures. Thus, simulation data can be acquired by massive quantities to fill the gap, despite their relatively "low quality" due to inherent model misspeciiiication or simplifica- tion. The low-quality simulation data selected represent ternary-septenary systems from an eleven-element palette consisting of mostly refractory elements (Al-Cr-Fe-HfMoo-Nb-- W-Zr). While there are existing experimental databases (Borg et al., 2020) and models to predict high-temperature yield strength in HEAs (Maresca & Curtin 2020). to our best knowledge, this is the first multi-fidelity dataset in the public domain that combines real experimental measure- ments and large quantities (over 100K) of simulation data for mechanical property prediction in HEAs. This specialized data should be able to predict high-temperature yield strength across broad range of HEAs. The predictions of this model could be used to pinpoint which alloys are the strongest at elevated temperatures, allowing experiments to focus on pre-sorted candidates for future study eliminating the need spend several weeks testing candidate without promise.
0.539
(Dfeomell
Composition, Temperature MoNbTaTi, 15K
Simulation Outcomes
Experiment Realls
0.122
0,539 <1K resultss
MoNb11T1273.55
0.475
>100K results
MoNI-TATT.222 15K MoNb 155.
0.14
0.927
MoNiTTTT 1273.5K
Pretraining
Fine-tuning
Sparsity noise robustness (learning from massive "low-quality" simulation data)
Sparsity for data-efficiency (taanferring to few- shot qualliyualtty experimental data)
Figure Proposed two-stage workflow. The HEA yield strength prediction model is first pre- trained on massive "low "low-quality" simulation data, and is then fine-tunede reee--teedtraasfereed on few-shot "high-quality" experimental data to optimize its prediction this target domain. Note that the tool of sparsity will be leveraged in both pre-training and fine-tuning stages, for the purposes of gaining noise cobustnss/rrassfrrallitt and enhancing data efficiency, respectively.
The new X- Yield benchmark is set to facilitate ML for HEA yield strength prediction, but learn- ing from such a multi-fidelity dataset is highly non-trivial. To this end, we next conceptualize a cross-quality few-shot transfer workflow: first pre-training the prediction model on the data-rich yet "low-quality" source domain (simulated data), and then fine-tuning the model towards the data- scarce yet "high-quality" target domain (experimental data). However, this vamilla workflow is chal- lenged by two issues: a significant quality gap between source and target domains, and an extreme data scarcity of target data. Inspired by the recent success of sparsity regularizers, we propose
2

---

Under review as a conference paper at ICLR 2023
incorporate sparsity to regularize both stages: sparsifying pee-trrreenan trainini improve the robustness and
cross-domain transferability of learned features (Guo et al. 2018 |ehwwgg et al. 2019| Chen et al.
2022 Sehwag et al., [2200] Ding al.
2022
Diffenderter al. 2021), , and sparsifying fine-tuning
to overcome data shortfalls (Liu et al.
2020; Chen et al., 2021] | Tao et al. (022). We demonstrate
proof-of-conepp experiments that even the simplest magnitude- baaud weight pruning could play effective regularization roles in our workflow. Furthermore, to avoid the ad-hoc two-step pruning as well as trial-and-error sparsity ratio selection at either stage, we propose a novel integrated opti- mization framework termed Bi-Level Regularized Pre- training and Transfer (Bi-RPT), that jointly learns optimal sparse masks and automatically allocates sparsity levels for both stages.
Our main contributions are summarized as follows:
We present X-Yie,,, the first public large-scale, multi-quality material science bench- mark for HEA yield strength prediction, containing alloys' compositions, processing tempera- tures, and yield strengths. Specifically, the yield strengths of 240 HEAs are experimentally mea- sured, while that of the remaining samples (over 100K) is calculated by simulations.
Dataset:
Methodology: we formulate a cross- gross-quality few-shot transfer workflow that can jointly exploit the simulated and experimental data for accurate predictions, and we innovate to leverage sparsity for addressing both the cimuuattddexperimmnntal domain gap and the scarcity of experimental data. While ad-hoe magnitude- based weight pruning is already found to be helpful, we further formulate an integrated bi-level optimization framework called Bi-RPT to automate the optimal sparse mask generation and sparsity ratio allocation at both pre-training and fine-tuning stages.
Extensive experiments show that Bi-RPT can boost performance on the X-Yield bench- mark alongside other synthesized testbeds. In particular, for the yield strength regression task, we achieve reduction of 19 ~ 38% on the test mean squared error by merely using 5-10% of the available experimental data. For the yield strength classification task, we achieve 0.98% ~ 1.53% of improvement in terms of the test accuracy.
Results:
2 RELATED WORK
2.1 MACHINE LEARNING IN MATERIALS RESEARCH
ML has been applied to solve a wide range of problems in materials science ranging from the fields of inorganic chemistry (Kailkhura al. 2019) sustainability (Gomes al. 2021), and metal- lurgy (Stan et al., 2020). with the typical purposes to predict materials properties and accelerate simulations (Pilania (02T). In both cases, MI. techniques hailed as reducing computational time in contrast to traditional materials science methods and are typically fast to develop (Wei et al.) 2019). Later on, deep learning has been successfully applied problems in the field of HEAs, in particular to predict phase formation (Lee et al. 2021b: b: Zhu et al. 2022). These approaches provide significant increases speed compared to phase predictions with CALculation of PHAse Diagrams (CALPHAD) (Saunders & Miodownik 11988, density functional theory (Parr 1983). and molecular dynamics methods (Shuichi] 1991) commonly used in materials science. Other properties predicted with deep learning are crystal structures, elastic constants (Liu et (023) and hardness (Bhandari ct al. 2021b). When comes specifically to the yield strength of HEAs, its prediction has also been previously explored with deep learning (Liu et al. 2023 Bhandari et al. (021a). However, major- ity of these efforts are restricted to the development of specific alloys (Zheng et al. 2021 | Bhandari et al. 2021bb or consist solely of transition metals (Wen al., 2019). and many studies also only use a small experimental dataset for prediction (Wen et al. [002)). I.. A generalized multi-fidelity ML. model predict yield HEA strength scale remains to be absent yet highly demanded.
2.2 SPARSITY REGULARIZATIN IN DEEP LEARNING
Sparsity or pruning was traditionally treated as a mainstream model compression approach in deep learning (Han et al. 2015). Recently, sparse regularizers have been increasingly used to enhance deep model robustness to various noise, malicious attacks, and distribution shifts. Guo et al. (2018); Sehwagg al. (2019); Gui et al. (2019) studied the intrinsic relationship between pruning and adver- sarial robustness. Recently, Diffenderfer et al.] (202T) comprehensively demonstrated the benefit of model sparsification to improve robustness distributional shifts (Hendrycks & Dietterich 2019
3

---

Under review as a conference paper at ICLR 2023
Bulusu et al. 2020). Sparse regularizers also exhibit promise in improving data efficiency. For ample, Zheng et al. (2019); Liu et al. (2020) proposed to learn model pruning strategies for feo-shot learning; Tian et al. (2020) combined model sparsification with meta-learning improve few-shot performance. Sparse regularizers have even been proven effective beyond few-shot image classifi-
cation, such as enhancing the data efficiency in image generation (Chen al. 202TT)
2.3 BI-LEVEL OPTIMIZATION
Bi-level optimization is a hierarchical framework where the variables in the upper-level optimiza- tion problem are dependent on the lower-level problem. Finn et al. (2017); |aaeewaarnn et al.) (2019) formulated the meta-learning problem in the form of bi-level optimization, and solve it by using first-order approximations. Other applications of bi-level optimization include data and label poi-
soning (Mehra et al. 2021] Huang et al. (020), and adversarial training (Zhang et al. (2021). In this work, we utilize bi-level optimization formulate our two-stage workflow with sparsification and find each stage's 's optimal weights and sparse masks while considering their sequential dependency.
3 X-YIELD: A NEW BENCHMARK FOR HEA YIELD STRENGTH PREDICTION
Overview Conventional alloys typically have one principal element with small amounts of other elements added to improve material properties (Ye et al. 2016) while HEAs can have multiple prin- cipal elements. The discovery of HEAs opened the door to a significantly wider range of design space to explore. most of which has yet to be examined (Miracle & Senkov] 2017). To address the task of using ML. to predict HEA yield strength, we focus on the sub-field of refractory HEAs (RHEAs). These materials have been demonstrated to maintain excellent mechanical properties at high temperatures (Li et al.] 2020). making them ideal candidates for hypersonics and aerospace industry applications. Prior work adopting ML to predict RHEA properties either uses solely exper- imental data (Wen et al.] (02T), or restricts predictions to only transition metals (Wen et al. 2019) or specific alloys such as MoN (Bhandari et al. 202 Hence, a generalizable ML prediction model for broad range of RHEAs is still absent. As mentioned carlier, it is impractical to generate high quantities of experimental data, especially for capturing yield strength at elevated temperatures. There are also challenges specific to high-temperattre measurements such controlling oxidation, confirming the heating profile and gradient within the samples, and use of more challenging experi- mental techniques (crosshead displacement) than those at lower temperatures (extensometers).
This work develops X-Yield, the first publicly available, multi-fidelity dataset consisting of over 100K low-quality simulated points and 240 experimental data points explore the RHEA design space. In this study alone, the entire composition space of all alloys containing between ternary- septenary systems from the Al-Cr-Fe-HH-Mo-ND-- -a----------------- family is examined. Since obtain- ing real high-temperatue strength data in the literature was taken close to room temperature (Borg et al.) (2020) even though there is more interest in RHEA properties at the high-temperature end (Miracle & Senkov| (017). From X- Yield, a multi-fidelity ML model is expected to be trained to predict high-temperature yield strength for a broad palette of RHEAS. The combination of nighttemperature yield strengths from the simulated dataset and experimental input can generate an ML model to accurately and efficiently predict tegperrumper yield strengths of alloys not included in the training set.
yield strength data is challenging, majority of the experimental yield
Dataset Construction The yield strength of the simulation data was predicted using the analytic and parameter-free mechanistic yield strength model developed by Maresca & Curtin (020). This model describes body-centered cubic (BCC) multi-principal element alloy (MPEA) solid solution strengthening associated with edge dislocations, in terms of elemental atomic volumes and clas- tic moduli. The yield strength was predicted for all ternary (1% increments), quaternary (1% in- crements), quinary (5% increments), senary (5% increments), and septenary (5% increments) al- loys from the 2500K in increments of 100K. This resulted in over three billion data points of which approximately 100, 000 were randomly selected for inclusion in this study. Note that even this advanced simula- tion model suffers from notable oversimplification and data quality issues. For example, The phase stability and dislocation character were not used to filter alloys in the study and the model may overpredict the yield strength of alloys with non non-BCC phases and underpredict the yield strength of alloys with different dislocation character, e.g. screw.
V--Z element family at temperatures between 300K-
4

---

Under review as a conference paper at ICLR 2023
0.5
Experimental Simuhsion
Experimental Simulation
2.0-
0.6
le-00-
Simulation (GPs)
1.5-
Density
Density
0.4
1.0-
0.2
0.5-
0.0
100
0.0
0.0
0.3
1.0
1.5
2/0
300
1000
1500
2000
2500
0.0
13
2.0
Yield Stress Value (GPa)
Temperature (K)
Experimental (GPa)
Figure 2: Left: the distribution of the yield stress; Middle: the distribution of the temperature; Right: pairwise visualization of the yield stress.
The highhhguhheeeee -uality experimental dataset was carefully filtered and curated from the database generated by Borg et al. (2020) consisting of mechanical property information for MPEAs. All data points were extracted that consisted solely of elements from the above element family, consisted only of BCC phases, were at temperatures higher than C0"C, and contained yield strength value.
Dataset Characteristics and "Quality Gap" As depicted in Figure 2 the simulation and exper- imental yield stress have different distributions. In the low-quality simulation data, considerable portion of yield stress annotations is greater than 2, while the experimental data hardly contains yield stress points beyond 2 (with one datapoint exception) due to the experimental condition con- straints. The distribution of the simulated yield stress is also significantly more skewed than the experimental ones. Pairwise visualization of the yield stress the 240 high-quality experimental samples suggests a substantial deviation between the simulation and experimental results. The dis- tributions of the processing temperatures are also heterogeneous, i.e., the simulation data presents uniform pattern while the temperatures in the conducted experiments are in a bimodal shape. These observations showcase the domain shifts "quality gap" between simulations and experiments.
CROSS QUALITY FEW-SHOT TRANSFER: A TWO-STAGE WORKFLOW AIDED BY SPARSITY (TWICE)
In this section, we first introduce the basic two-sage workflow, upon which we propose sparsification methods vanilla approach "Hand-Tune" and an improved principled framework "Bi-RPT").
Basic Two-Stage Workflow: Pre-training then Fine-tuning Let us denote the high- quulity target domain (experimental data) by D., and the low-quality source domain (simulated data) by D., Our goal is to learn generalizable predictor over D, while leveraging the aid of D... One naive idea is to simply combine the two data domains and jointly train a supervised model. However, the large domain gap between D. and Dr, as well the sample scarcity in D,, will result in the jointly trained predictor to fit DD poorly. Instead, we propose to formulate our workflow as a two-stage pipeline: first pre-training a model on D., and then fine-tuning to optimize the prediction over D,..
Incorporating Bi-Stage Sparsity: A Vanilla Approach. However, the features learned from D, will inevitably suffer from domain gap and noise when applied towards D., and the extreme data scarcity of D; remains as another challenge. Inspired by the recent success of sparse regularizers improving both robustnessrransferabbii and data efficiency, we attempt incorporate sparsity into both stages to address the two-fold challenges.
We first prove our concepts by proposing a vanilla ad ad-hoc approach, which we refer to as Hand- Time. Starting from pre- pre-training over D., we perform the standard iterative magnitude pruning (IMP) (Frankle & Carbin 2019) during pre-training. particular, we alternate between (re-)training and pruning; each time, we prune the 20% smallest-magnitue weights by default and continue (re-) (re-)training the remaining non-zero weights. Such a "prune-and- retrain" routine is repeated for N, rounds obtain the final sparse mask m. (1 denotes the element to be non and 0 to be pruned) associated with the pretrained model weight. Then, we move on to fine-tuning over D,, and start another round of IMP top of the pre-trained model: note that this second-stage IMP continues only on the subset of current non-zen weights, ie.. the 1-valued regions in m.. IMP in fine-tuning repeats another N, round (with the identical protocol the first
weights from the existing non-zero
5

---

Under review as a conference paper at ICLR 2023
stage), yielding another sparse mask m,- The final model uses the joint sparse mask m, C m where @ represents the point-wiid wise product.
Hereby, Ne and N; are hyperparameters that control the sparsity allocation between two stages. Intuitively, while certain sparsity may contribute to noise resilience, an overly large N, will cause the pre-trained model to be over-sparsified, limiting its capacity to learn sufficiently informative and transferable features. Fine-tuning has a similar trade-off. Therefore, N₂ and N, have to be manually tuned for the two-stage workflow to achieve good performance (see Appendix (.2).
Principled Bi-Stage Sparsity Integration with Bi-RPT. Hand-Tune has some apparent flaws: (1) it removes weight elements by merely using weight magnitude information, which is explicitly task-driven; (2) the two sparse masks m, and m. are decided in a sequential manner rather than jointly optimized, e.g., learning my will passively suffer from any artifact in learning m.; (3) the sparsity ratios assigned in both stages, as controlled by N₂, and Nr, need to be manually tuned, without any obvious insight beyond exhaustive hyperparameter search.
We, therefore, devise a more principled framework that can jointly learn the optimal sparse masks as well as sparsity allocations for both stages, termed Bi-Level Regularized Pre-training and Trunsfer (Bi-RPT). The optimization problem is expressed as follows (y is a coefficient):
min 0. m.m, m, (m
E [m., ④ m.) C 0 Te. Yo 10" mm)] + ? (R(m,, O m.,)
(1)
s.t. 10 m
arg min 0,m, (m.:Bo)~D,
E (m,, ① 0.
where L,/C; represents the objective functions for the two stages, respectively, 0 represents the models' parameters, and R represents the sparsity regularizer. Seemingly complicated at the first glance, the bi-level optimization formulation of Bi-RPT actually admits a clear physics "workflow" interpretation. Let us start from the lower-level problem ( which instantiates the sparsity regular- ized pre-training stage over D.: its outputs include the pre-trained weight 0" and the corresponding sparse mask m. Then, the upper-level problem (T) depicts the sparsity regularized fine-tuning over D. which inherits both 0" and m as its starting point. It continues to modify the weight well as to evolve another sparse mask m,. Eventually, a total sparsity over the joint mask m. my and the final model weights could be represented as (m, ④ m.) @ 0.
function R enforces the
Importantly, the lower- and upper-level problems in Bi-RPT are solved in end-to-end manner, meaning that even the fine-tuning depends on 0 and m it can, in turn, provide feedbacks for adjusting the latter: hence synergistic optimization is achieved between two stages. The sparse mask selection now directly hinges on the end task (target domain loss Lx) rather than heuristics such weight magnitudes. Lastly, the sparsity levels of m, and m, do not need to be separately designated nor manually controlled: we automatically learn the sparsity ratio allocation, under only the total sparsity regularizer R.
To practically solve the bi-level optimization of Bi-RPT, we derive algorithms whose details can be found in Appendix A For the sparsity regularizer R. we adopt the smoothed by term (Guo et al.] 2021 to facilitate differentiable training: a gate function gx almost binary when the e is small, is used. In general, for the lower-level optimization problem, we update the models' parameters 0 by gradients to minimize L.: for the upper-level optimization, we utilize the gradient unrolling to develop update rules for
x² /2x22 + +c), whose outputs are
4.1
EXPERIMENTS ON IMAGE DATA
For proof-of-concept, we conduct experiments a synthesized testbed of image classification, to compare Hand-Tune and Bi-RPT. We adopt two source-domain dataset options: ImageNet (Deng et al. 2009) and ImageNet-C (Hendrycks & Dietterich] 2019), the latter more noisy and corrupted. Two target-domain options are also accompanied: CUB-200 (Wah et al., 201I) and CUB-200 (10- shor), the latter designed to be rigorously "few-shot" where each class has only 10 training samples. Different combinations of D., allow us to conduct controlled experiments for stretch-testing various algorithm options' noise robustness well data efficiency.
Several baselines are compared to Hand-Tune and Bi-RPT: (1) Pretrain- and-transfer the basic workflow of pre-training on D, followed by finetuning on Dr, with sparsity involved; (2) Pretrain

---

Under review as a conference paper at ICLR 2023
sparsity only /transfer sparsity only: following our proposed vectrai--nnd-traae workflow, but conducting IMP to only the ure-trainin/finetuminn stage; (3) No Pretraining: directly training on Dr without using Dx; (4) Mix Training: training one model on D, and D, combined. For those methods with IMP involved, we hand-select the sparsity ratio(s) for either or both stages that yield the highest generalization performance on Dr, i.e., from hyperparameter grid search via cross-validation.
Table D reports the accuracies of all methods over various source/target combinations, on the same
testing set of CUB-200 in Table [ All methods use the same ResNet-18 18 backbone. We high- light several key observations: (1) incorporating D, in general helps both CUB-200 and CUB-200 (10-shot), and the improvement margin is much more substantial for the few-shot case; (2) models trained by Mix Training fail to generalize on Dr- in fact even worse than No Pretraining, showcas- ing the negative influence of the quality gap: (3) in the same regime of pre-training then fine-tuning, adding appropriate sparsity helps, and two-stage sparsity can help more; (4) Bi-RPT stably outper- forms Hand-Tune (especially, very notably in few-shot cases), despite the best efforts in tuning the latter's hyperparameters. More observations and analysis can be found in Appendix [ (Tables A5
A8 and Figure A4): including but unlimited to the backfiring effect of "over-sparsffication" and the compound influence of per-stage IMP sparsity allocation in Hand-Ture
Table 1: Experiments on image data: testing aceuracy of fine-tuned ResNet-18 on CUB-200/ CUB- 200 (10-shot) as D, after pretraining on ImageNet and ImageNet-C as D., respectively.
D,
D;
Methods
Two-stage
m,
m
ImageNet
ImageNet-C.
No Pretraining
x
X
X
44.27% 7.98%
Mix Training
x
X
X
30.88% / 7.7
27. 27.32% 86.
CUB-200
Pretrain-and siliin--nd-traae -rannsfferr
/
x
74.16% / 38.66% x
71.59% / 32.14%
/ CUB-200 (10-shot)
Pretrain sparsity only
V
x
76.01% 1.1 / 40.73%
73.70% / 38.76%
Transfer sparsity only
/
x
/
74.16% / 38, .090%
71.83% / 32.58%
Hand-Tume Tune
/
/
76.01% / 40.78%
74.01% 39.94%
Bi-RPT
78.60% /
51.55%
76.29 / 47.01
5
MAIN EXPERIMENTS ON THE X- YIELD BENCHMARK
5.1 IMPLEMENTATION DETAILS
Task Definition. The most naturally defined task on X-Yield is the regression, i.e., predicting the yield strength of alloys, and calculating the error between the model prediction and "ground-truth" (experimental results). Besides the regression task, we formulate another surrogate classification task by constructing five categorical labels based on the bin intervals where the ground-truth yields strength fall in. These intervals are: [0, 0.5), [0.5, 1). 1.5). [1.5, 2), and [2, co).
Data Representations. We featurize each HEA by mapping its composition and temperature into a "pseudoimage" (please refer to Appendix B.5 and Figure (55). The pseudoimages have two chan- nels: the first channel is constructed from the alloys' composition using the randomized periodic table structure (Feng et al. 2021). As the temperatures are originally recorded in Kelvin, we convert and normalize them by T nonmalized = (K- 273.15) (0..155)2000 /200 where K the temperature in Kelvin, and then embed the converted temperature as the second channel in pseudoimages.
Architectures and Baselines. The structure of the ML predictor we use a convolutional neural network. It consists of 3 convolutional layers, each of which has kernel size of 3, followed by multi-
Batch Normalization (loffe & Szegedy 2015) and ReLUU (Gloot et al., 201T) activation. layer perceptron is appended after the convolutional neural network to generate the final prediction for both the regression and classification tasks. We focus on comparing our main proposal, Bi-RPT, with two baselines of No Pretraining and Pretrain-and- transsfrrr.rrr.....e same defined in Section 4.1
Evaluation Metries and Data Splits. We evaluate each method in two ways. Besides the widely used 10-fold cross- validation, explore two challenging extreme few-shot settings: sample 5% (and 10%) of experimental data from each alloy type (ternary, quaternary, quinary and senary) as training set, and the rest left as the test set. Note that these classes are not the classification
7

---

Under review as a conference paper at ICLR 2023
labels. Eventually, we have only 23(1)) training samples and 217() testing samples. All the low-quality (simulated) data is used for pretraining where applicable. For the regression task, we report the best mean squared error (MSE) the test splits; and for the classification task, we report models' test split accuracy to measure their performance.
Training Hyperparameters We pretrain the ML predictor the simulation data for 10 epochs.
During the pretraining, we use the Adam optimizer (Kingma & Ba (014) with an initial learning rate of 1 X 10-4 and cosine annealing schedule (Loshchilov & Hutter 2016). For the transfer stage, we fine-tune the pretrained model on the experimental data for 90 epochs. The optimizer we use the SGD optimizer with an initial learning rate of x 10 10-3 We also decay the learning rate by 10 for every 30 epoch. The batch sizes for pretraining and fine-tuning are 16 and 4, respectively.
5.2 MAIN RESULTS
Classification and regression with extreme sho settings. We first apply Bi-RPT to solve the
regression and classification tasks under the two extreme few-shot settings where only 5% and 10% experimental data are available, respectively. Table 2 shows that: (1) pretraining on simulation data can benefit the ML predictor consistently both the regression (over 10% reduction in MSE) and classification (over 11% improvement in accuracy) tasks, especially when the data is more scarce; (2) the integration of sparsity into the pretraining and transfer workflow can further strengthen the predictor's ' generalization, improving accuracy by 0.98% and reducing MSE by 8.91% using merely 10% training experiment data, and the improvement also becomes even more significant with 5% training data (1.53% increase in terms of the accuracy and 19. 19.75% reduction in terms of the MSE).
Table 2: Test accuracy on the testing set of different splits of high-fidelity alloy data. The experi- ments are repeated 10 times, and report both the mean and the 95% confidence interval.
10% train samples
5% train samples
Method
Test MSE
Test Accuracy
Test MSE
Test Accuracy
No Pretraining
0.114 = 0.007
54.84 1.59%
0.212 + 0.041 0.162 + 0.002
47.25 + 0.80% 62.84 + 1.96%
Pretrain-and-transfer
0.101 + 0.001
65.85 + 0.88%
Bi-RPT
0.092 = 0.011
66.83 = 1.41%
0.130 = 0.006
64.37 = 1.10%
Classification and regression with 10-fold cross-validation. On the slightly "data-rich" 10-fold cross-validation setting, we have observed a similar trend: the bi-stage regime of pretraining and transfer out-performs the single-stage training pipeline, and incorporating sparsity can consistently provide remarkable improvement to the ML predictor, particularly in the regression performance.
Table 3: Classification and regression performance under the ten-folded cross-validation settings.
Classification
Regression
No Pretraining
67.50 = 5.16%
0.226 + 0.027
and-transfer
82.09 = 3.86%
0.206 0.026
Bi-RPT
82.50 + 2.93%
0.068 + 0.009
Performance comparison alloys at various temperatures. Based on the trained model with 10% experimental data, we predict the yield strength of three alloys, MoNbTaTi, MoNbTaTTW and HrMoNbTaTÍZı, at different temperatures. Table 4 shows the predicted yield stress on these three alloys using Bi-RPT and baselines. On the quinary and senary alloy systems, Bi-RPT shows ex- ceptional precision in predicting the experimental yield stress. More scrutiny of those predictions reveals several findings that neatly align with our material science expertise. For example, it is known that screw dislocations are more likely to be dominant than the edge in MoNbTi and NRТTТi ternaries (shown from the ternary comparison in the Citrine database (Borg et al.) 2020)). Thus it makes sense that the model under-predicts the MoNbTaTi and MoNbTaTiW cases: our model seems to correctly pick up these differences and predicts higher yield strength. Another example is that our model over-predice HfMoNbTa at lower temperatures (300K- ~ 900K). Since all our col- lected experimental samples are 100% body-centered cubic (which shows, admittedly, limitation
8

---

Under review as a conference paper at ICLR 2023
Method
NP
0.10
PT
MSE
Bi-RPT
0.05
0.00
1073.15
1273.15
1473.15 Temperature (K)
1673.11
1873.15
Figure 3: Prediction MSE under different temperatures. We compare the results of three methods: No Pretraining (NP), Pretrain-and-transie (PT), and Bi-RPT.
of Xield compared to the tremendous variations real-world HEAs), it is likely that non-BCC phase will appear at lower temperatures, hence lowering the yield strength.
Table 4: Predicted yield stress of different alloys under different temperatures. Only 10% of the experimental data are available during fine-tuning. We compare the predicted yield stress generated by Bi-RPT with our "No Pretraining" (NP) and "Pretrain-and aandtrrasserr (PT) baselines and the sim- ulation. The numbers with the smallest eπror are marked in bold.
Predicted Yield Stress (GPa)
Alloys
Temperature (K)
Experimental (GPa)
Bi-RPT
NP
PT
Simulation
293.15
1.078
1.170
1.062
0.475
1.210
473.15
0.965
1.004
0.902
0.381
0.868
MoNoTaTi
673.15
0.746
0.772
0.731
0.282
0.685
873.15
0.508
0.642
0.584
0.472
0.593
1273.15
0.425
0.570
0.488
0.114
0.539
298.15
1.268
1.031
1.068
0.814
1.399
873.15
0.677
0.569
0.607
0.372
0.689
MoNbTaTIW
1073.15
0.618
0.520
0.523
0.294
0.674
1273.15
0,536
0.530
0.486
0.232
0.620
296.15
1.527
1.051
1.132
1.849
1,515
873.15
0,861
0.556
0.685
1.178
0.973
HIMoNbTaTiZr
1073.15
0.762
0.536
0.612
0.516
0.791
1273.15
0.662
0.563
0.573
0.421
0.753
Performance at high temperatures. One of the important tasks in the alloy design community to find alloys that are capable of withstanding stress at high temperatures. To verify Bi-RPT can provide reliable recommendations to help the community achieve this goal, we look deeper into the predictive performance in high-temperature regimes. We train our model with 10% data, predict the yield stress for the rest 90%, and compare the predictive quality of models high temperatures in Figure 3 We can see that Bi-RPT significantly outperforms other baselines, especially at tempera- tures greater than 1400K. These results suggest Bi-RPT could serve strong tool for designing HEAS with superior yield stress at elevated temperatures.
CONCLUSIONS
To address the important yet challenging problem of HEA yield stress prediction, curated and released X- Yield, the first large-scale, multi-fidelit benchmark. To effectively leverage this bench- mark, we also designed a two-stage cross-quality few-shot transfer workflow and proposed utilize sparsity to tackle both challenges of low data quality at pretraining and scarcity at transfer. Besides ad-hoc methods, we formulated a principled bi-level optimization framework to automatically learn the optimal sparse masks and sparsity allocation between two stages. Extensive experiments image data testbeds and X- Xieil demonstrate the Bi-RPT showed a substantial improvement over closely working with material scientists to validate
both
existing baselines. Moving forward, our ML prediction results based on their domain expertise, and the team has already identified some alloy candidates that appear promising to be experimentally validated.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Gleeble. BtLLhttps:/// http://wwwwweeee.eoo
Accessed: 2022-09-26.
Uttam Bhandari, Md Rumman Rafi, Congyan Zhang, and Shizhong Yang. Yield strength prediction of high-entropy alloys using machine learning. Materials Today Communications, 26: 26:101871,
2021a.
Uttam Bhandari, Congyan Zhang, Congyuan Zeng, Shengmin Guo, Aashish Adhikari, and Shizhong Yang. Deep learning-basee based hardness prediction of novel refractory high-entropy alloys with perimental validation. Crystals, 1:46, 2021b.
Christopher KH Borg, Carolina Frey, Jasper Moh, Tresa M Pollock, Stéphane Gorsse, Daniel B Miracle, Oleg N Senkov, Bryce Meredig, and James E Saal. Expanded dataset of mechanical properties and observed phases of multi-princinn element alloys. Scientific Data, 7(1):1-6, 2020.
Saikiran Bulusu, Bhavya Kailkhura, Bo Li, Pramod Varshney, and Dawn Song. Anomalous example detection in deep learning: A survey. IEEE Access, 8:132330-132347, 2020.
Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Data-efficient gan train- ing beyond (just) augmentations: A lottery ticket perspective. Advances in Neural Information Processing Systems, 34:20941-20955. 2021.
Tianlong Chen, Zhenyu Zhang, Pengjun Wang, Santosh Balachandra, Haoyu Ma, Zehao Wang, and Zhangyang Wang. Sparsity winning twice: Better robust generaliztion from efficient training. arXiv preprint arXiv:2202.09844 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
James Diffenderfer, Brian Bartoldson, Shreya Chaganti, Jize Zhang, and Bhavya Kailkhura. A winning hand: Compressing deep networks can improve out-of-distribution robustness. Advances in Neural Information Processing Systems, 34, 2021.
Shaojin Ding, Tianlong Chen, and Zhangyang Wang. Audio lottery: Speech recognition made light- noise-robust, and transferable. In International Conference on Learning Repre- sentations, 2022. URL https: / / forum?idd 9NN.
Shuo Feng, Huadong Fu, Huiyu Zhou, Yuan Wu, Zhaoping Lu, and Hongbiao Dong. general and transferable deep learning framework for predicting phase formation in materials. npj Computa- tional Materials, 7(1):1-10, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1 1135. PMLR, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: networks. In International Conference on Learning Representations, 2019.
Finding sparse, trainable neural
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourleenth international conference on artificial intelligence and statistics, pp- 315-323, JMLR Workshop and Conference Proceedings, 2011.
Carla P Gomes, Daniel Fink, R Bruce van Dover, and John M Gregoire. Computational sustainability meets materials science. Nature Reviews Materials, 2021:645-644 2021.
Shupeng Gui, Haotao Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu. Model com- pression with adversarial robustness: A unified optimization framework. Advances in Neural Information Processing Systems. 32, 2019.
Yi Guo, Huan Yuan, Jianchao Tan, Zhangyang Wang, Sen Yang, and Ji Liu. Gdp: Stabilized neural network pruming via gates with differentiable polarization. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5239-5250, 2021.
10

---

Under review as a conference paper at ICLR 2023
Yiwen Guo, Chao Zhang, Changshui Zhang. and Yurong Chen. Sparse dnns with improved adver- sarial robustness. Advances in neural information processing systems, 31, 2018.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 150.0015,,,0 2015.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. 12261, 2019.
W Ronny Huang. Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Prac- tical general-purpose
purpose clean-label data poisoning. Advances in Neural Information Processing Systems, 33:1 12080-1211,, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Bhavya Kailkhura, Brian Gallagher, Sookyung Kim, Anna Hiszpanski, and T Han. Reliable and explainable machine-learning methods for accelerated material discovery. npj Computational Materials, 5(1):1 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Chanho Lee, Francesco Maresca, Rui Feng, Yi Chou, T Ungar, Michael Widom, Ke An, Jonathan D Poplawsky, Yi-Chia Chou, Peter K Liaw, et al. Strength can be controlled by edge dislocations in refractory high-entropy alloys. Nature communications, 12(1):1-8, 2021a.
Soo Young Lee, Scokyeong Byeon, Hyoung Scop Kim, Hyungyu Jin, and Seungchul Lec. Deep learning-based phase prediction of high entropy alloys: Optomization, generation, and explana- tion. Materials & Design, 197:109260, 2021b.
Xiang-Guo Li, Chi Chen, Hui Zheng, Yunxing Zuo, and Shyue Ping Ong. Complex strengthening mechanisms in the nbmotaw multi-principal element alloy. npj Computational Materials, 6(1): 1-10, 2020.
Jinbao Lin, Weijie Ren, Qudong Wang, Lifeng Ma, and Yongjun Chen. Influence of grain size and texture on the yield strength of mg alloys processed by severe plastic deformation. Advances in Materials Science and Engineering, 2014.
Chen Liu, Chengming Xu, Yikai Wang. Li Zhang, and Yanwei Fu. An embarrassingly simple baseline to one-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops. pp. 922-923, 2020.
Xiangliu Liu, Jiaxin Zhang, and Zongrui Pei. Machine learning for high-entropy alloys: Progress,
challenges and opportunities. Progress in Materials Science, 101:101018, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint /60i 1608.03833 2016.
Francesco Maresca and William Curtin. Mechanistic origin of high strength in refractory bec high entropy alloys up to 1900k. Acta Materialia, 182:235-249, 2020.
Akshay Mehra, Bhavya Kailkhura, Pin-Ye Yu Chen, and Jihun Hamm. Understanding the limits of unsupervised domain adaptation via data poisoning. Advances in Neural Information Processing Systems, 34, 2021.
Daniel B Miracle and Oleg N Senkov. A critical review of high entropy alloys and related concepts. Acta Materialia, 12 122:448-511, 2017.
11

---

Under review as a conference paper at ICLR 2023
Atsushi Nitanda. Stochastic proximal gradient descent with acceleration techniques. Advances in Neural Information Processing Systems, 27, 2014.
Robert G Parr. Density functional theory. Annual Review of Physical Chemistry, 34(1):631-656 1983.
Ghanshyam Pilania. Machine learning in materials science: From explainable predictions to tonomous design. Computational Materials Science, 1021:1103600 2021.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with im- plicit gradients. Advances in neural information processing systems, 32, 2019.
Nigel Saunders and A Peter Miodownik. CALPHAD (calculation of phase diagrams): a compre- hensive guide. Elsevier, 1998.
Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. Towards compact and robust deep neural networks. arXiv preprint arXiv: 1906,0669 10, 2019.
Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. Hydra: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems, 33:19655-19666, 2020.
Nosé Shuichi. Constant temperature molecular dynamics methods. Progress of Theoretical Physics Supplement, 103:1-46 1991.
Tiberiu Stan, Zachary T' Thompson, and Peter W Voorhees. Optimizing convolutional neural net- works perform semantic segmentation large materials imaging datasets: X-ray tomography and serial sectioning. Materials Characterization, 2020: 100119, 2020.
Ran Tao, Han Zhang, Yutong Zheng, and Marios Savvides. Powering finetuning for few-shot learn- ing: Domain-agnostic bias reduction with selected sampling. 2022.
Hongduan Tian, Bo Liu, Xiao- Tong Yuan, and Qingshan Liu. Meta-learning with network pruning. In European Conference on Computer Vision, pp. 675-700. Springer, 2020.
Issac Toda-Carballo, Enrique Galindo-Nava, Rivera-Díar del Castillo, and Pedro. Understand- ing the factors influencing yield strength on mg alloys. Acta Materialia, 2014.
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 969-977, 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Jing Wei, Xuan Chu, Xiang-Yt Yu Sun, Kun Xu, Hui-Xiong Deng, Jigen Chen, Zhongming Wei, and Ming Lei. Machine learning in materials science. InfoMat, 2019:338-358, 2019.
Cheng Wen, Yan Zhang, Changxin Wang, Dezhen Xue, Yang Bai, Stoichko Antonov, Lanhong Dai, Turab Lookman, and Yanjing Su. Machine learning assisted design of high entropy alloys with desired property. Acta Materialia, 170:109-117 2019.
Cheng Wen, Changxin Wang, Yan Zhang, Stoichko Antonov, Dezhen Xue, Turab Lookman, and Yanjing Su. Modeling solid solution strengthening in high entropy alloys using machine learning. Acta Materialia, 212:1 16917, 2021.
YF Ye, Qing Wang, Jiatian Lu, CT Liu, and Yancong Yang. High-entropy alloy: challenges and prospects. Materials Today, 1966)344-366 2016.
J-W Yeh, S-K Chen, S-J Lin, J-Y Gan, T-S Chin, T-T Shun, C-H Tsau, and S-Y Chang. Nanos- tructured high-entropy alloys with multiple principal elements: novel alloy design concepts and outcomes. Advanced engineering materials, 6(5):299-303. 2004.
12

---

Under review as a conference paper at ICLR 2023
Yihua Zhang, Guanhuan Zhang, Prashant Khanduri, Mingyi Hong, Shiyu Chang, and Sijia Liu. Revisiting and advancing fast adversarial training through the lens of bi-level optimization. arXiv preprint arXiv:2112.12376, 2021.
Tao Zheng, Xiaobing Hu, Feng He, Qingfeng Wu, Bin Han, Chen Da, Junjie Li, Zhijun Wang, Jincheng Wang, Ji-jung Kai, Zhenhai Xia, and Liu C T. Tailoring nanoprecipitates for ultra- strong high-entropy alloys via machine learning and prestrain aging. Journal of Materials Science & Technology, 69:156-167, 2021.
Yin-Dong Zheng, Yan-Tao Ma, Ruo-Ze Liu, and Tong Lu. A novel group-aware pruning method for few-shot learning. 2019 International Joint Conference on Neural Networks (IJCNN) pp- 1-7, 2019. doi: 10.1 2019
Wenhan Zhu, Wenyi Huo, Shiqi Wang, Xu Wang, Kai Ren, Shuyong Tan, Feng Fang, Zonghan Xie, and Jianqing Jiang. Phase formation prediction of high-entropy alloys: a deep learning study. Journal of Materials Research and Technology, 18:800-809, 2022.
13

---

Under review as a conference paper at ICLR 2023
A MORE DETAILS ON METHODS
In this section, we present the technical details of our proposed method and framework ("Hand- Tune" and "Bi-RPT").
A.1 HAND-TUNE
Hand-Tune decides sparse masks for the two stages in an iterative way as explained in Algorithm
Algorithm Hand-Tune
Input:
Initialization weights dg. low-quality pretraining dataset D.,, high-quality fine-tuning dataset D,, number of IMP rounds N, for the pretraining stage and N; for the fine-tuning stage. Output: the trained weights 0", the sparse mask m, for the pretraining stage, and the sparse mask m, for the fine-tuning stage.
Initialize the sparse masks m, for the pretraining stage to be a all "1" mask.
Initialize the model's weight 0, and train the weights on D, to obtain 0,.
D IMP at the pre-training stage Prune 20% of the smallest-mage magnitude weights from the non-zero regions of m, 0,, by setting the values at corresponding positions to those weights in m. to "0". (Re-)train the sparse weights m, end for
for i = 1, 2,2,..., N₂ do
D.. Only is updated.
Initialize the sparse masks the fine-tuning stage my to all "1" masks and freeze m.. Initialize model's 's weight as m, @ 0.. and train on D, to obtain m, @ t for 1,2,..., N, do
IMP at the fine-tuning stage
Prune 20% of the smallest-magninee mi) @ 2.. by setting the values at corresponding positions to those weights in m, to "0".
weights from the non-zero regions of weights (m, ?
(Re-)train the sparse weights (m, m.) O 0, on D, Only O, is updated. end for
Obtain the final sparse weights (m, O m.) C 0 and return 0* m, and my.
A.2 B1-RPT
We now build the techniques to solve the bi-level optimization problem formulated in Bi-RPT.
Formulation
min 0.m, my (m..
E
[Ce((m,
@ m.) 3 0, Tt, vr.ye 0o m + <R(m, @ m,)
s.t. {0", m = arg min 0.m,
E (w..
[. ① V.).
Lower-level problem We solve the lower-level problem through a p-step SGD unrolling. Let (*) (k) be the mask for the pretraining stage. The superscript (k) indicates
be the model weights, and m they have been updated on the upper-level for k steps.
((k) and m (k) will be the starting points for the lower-level optimization problem. o(² and m 2. are weights and mask, respectively, after being updated for steps on the lower-level optimization problem (implying ((0) = (()) and m
(0) m ). The update rules can be written as
(k)).
0 0 0 0(P)) 0..- 1) VoL 0-0(--1)
(3)
(0) (k) (0) m
(x-1) Am, Vm m= (x-1)),
(4)
where is the learning rate for the model weight 0, and Am, the learning rate for the mask m. the lower-level optimization problem.
(6)
A14

---

Under review as a conference paper at ICLR 2023
Upper-level problem and Sparse Regularization Loss The upper-level problem is the sum of two losses: a normal training loss L2 and a sparse regularization loss R (7 is coefficient).
We first develop update rules for the training loss L. The weights 0 (= ( 0,1)) and masks m (P) from the lower-level problem after unroll steps will serve as the initialization of the upper-
m s,l level problem. We update the model weight 0 and masks at the upper level by applying gradient- based methods (take SGD as example):
dL
g(k+1) -0-
A d0=
(5)
θc.
0C. 0m,
0- Am 00
dmg 00
where Au is the learning rate for the weights for the upper-level optimization problem. The gradient
on my is easy enough: ōm,,"
dc. while the gradient on m, is slightly complicated:
dC,
OCt dC. 00
(6)
dm;
0m;
a0- dm:
We expand the latter terms in Eqn. and Eqn. based on the first-order approximation (picking p 1) on the lower-level problem:
00
0(0(0) voVo)
0(0(00 VoC.)
00(0)
+
0m: 8(m
(0) Am,1 V Vm,Ca)
00(0)
0(m
(0)
Am, V V.,
0 0)) AoVoC.)
dm
(0)
s.l
8m
0(m s,1
(0) Am. V m. Co)
(1 - VV.L))( Am. V om,
(-), V m. C.) Am, V VmmLLo La) -1,
0m; 8(m Am,1 Vn..Va Vn, (m,fs) fss)
(I -V2 o
00- 00 0))
VoL.,)
(-N,Vm,CC.--11 Am. V V2m Cz).
(8)
Further approximations can be made to avoid the matrix inverse and save computation:
00 ~ - -
dm: 00
~ -Am V vm,
dm;
Based the rules, m, and m, be optimized by:
m, oc. (3) m (k+1)
OC,
dC, V m. oLe m
(k+1) mi
(R) Am
mk Am 0m, 00
+> >
Ōm,
where the superscript (k) means the steps updated.
We then focus on the latter term. We choose En loss (i.e. the number of non-zero elements) as the sparse regularizer R, which is not differentiable and difficult to optimize. Therefore, follow Guo et al. (2021) to use the smoothed to formulation to facilitate differentiable training. Specifically, gate function (x: . where is small positive number, is used replace the binary masks, which instead parameterized by ge (m.) and ge (m:). We decay the value of every
=
A15

---

Under review as a conference paper at ICLR 2023
epoch, and the gate function will gradually output only polarized numbers (i.e., 0 and 1). We further apply the proximal-SGD (Nitanda 2014) to minimize the to loss: after we update the m, and m, with respect to C, by gradient descent-baset methods (Eqn. [), we use the proximal operator to alternatively update each mask. For m,, the formulation can be written as:
prox yR (m!*+1)) arg min 2 Il
(++ (++++++))) Dom! (k+1)
(1+++))) oom (k+ (
11 12 Am C++ ! (k+1) Ioo
We follow (Guo al. 2021) solve it by relaxing the & norm problem, which has closed form solution:
8,6
(k+1)
7Am
mat
(M+1)
2
7mm m [1
m
m a,3
(k+1)
+
mi (+41)
Jam
magi
(k+1)
7mm S i (8+1)
(10)
0,
7m
13
(l+1)
<
t
7mm
1k+1]
where Ma, is the i-th element in m. (the same for mz,)). Similarly, we derive the update for m.:
(k+1)
i m (
7mm
mi
(k+1)
2
m
m t,
(l++)
mi
(k+1)
+
th
my,
(k+1)
S
7m
(1+1)
(11)
72m
(k+1)
7mm
m
Finally, we combine all these components into Algorithm
Algorithm Solving Bi-RPT
Input:
Initialization weights Go. training loss functions for two stages C. and Le, low-quality pretraining dataset D.,, high-quality fine-tuning dataset De, number of steps for gradient unroll p. Output: Trained model weights 6, sparse masks m, and m,. Train 0 on D. get weights 0.
while not converged do
Given the fixed m. update the weights @ Dz by gradient unrolling (Eqn. Update the weights 0 by Eqn. Update the masks m, and m, by Eqn. 9
Update the masks m. and m. by Eqn. 10 and Eqn. 11 end while
B MORE EXPERIMENTS DETAILS AND RESULTS
B. BASELINES AND HYPERPARAMETERR
We list the hyper-parameters we used for all the baselines in this section.
General Settings. When pre-training the models on D, (ImageNet and ImageNet-C), we use the SGD optimizer and a learning rate is 4 x 10-1 We linearly warm-up the learning rate within 5 epochs, and then decay it by 10 for every 30 epochs. Models are pretrained for 95 epochs on D.. with batch size of 1024. On Dr, i.e., CUB-200 and CUB-200 (10-shot), we set the initial learning rate as x 10 10-3 The learning rate is decayed by 10 every 30 epochs, and the model is trained for 90 epochs with batch size of 64.
For Hand-Tune Tune, we train the models with 95 epochs from scratch on D, to get a densely pretrained models. The number of training epochs is reduced to 45 after the pretrained model is derived. After the pretraining stage ends, we continue to transfer the model on D, following the above hyper- parameters. The number of epochs is also reduced to 45 after we prune the weights
For No-Pretraining, we train the model using an initial learning rate of 1 x 10-2 2 and a batch size of 64. For Mix- Mix-Training, as the number of classes is different for ImageNet and CUB-200,
A16

---

Under review as a conference paper at ICLR 2023
two fully- tulne-connected layers top the normal ResNet-18 backbone, and train them simultaneously. We sample batches from the two domains (D, and D:) using the same batch size 64. The initial learning rate for these methods are 1 10 and it is decayed by 10 every 30 epochs.
For Bi-RPT, we follow the same learning rate settings despite some additional newly introduced. The learning rate for the lower-level problem (Ar) is 1 10-3, the same as the learning rate for upper-level problem (A,)) The value of 7 is set to x 10-4, which are determined through ablation studies in Table A10 The value of Am are set 3.5, which are also determined through ablation studies in Table All
are
B.2 PERFORMANCE HAND-TURE UNDER DIFFERENT LEVELS OF SPARSITY
We report the performance of the Hand-Tune method under different levels of sparsity. We con- duet experiments with N, = {0. 1, 2. 3, 4, 5) and N, = {0. 1,2, 3, 3.4), resulting sparsity levels pre-training stage of (0.00%, 20.00%, 36.00% 48.80%, 59.04%, 67. 67.23%) and sparsity levels transfer stage of {0.00%, 20.00%, 36.00%, 48. 88.80% 59.04%). We conduct experiments over all the combinations of pretraining and transfer pruning rounds. More specifically, on D. for N₂ rounds, and continue to perform IMP on Dr for another N₂ rounds. The experiment results over various souree and target combinations are shown Table A5 to Table A8 Note that all the models are evaluated on the testing samples in D,,
first perform IMP
From this series of tables we observe that: (1) sparsity at pretraining helps improve the model's performance on 'D after fine-tuning, and the performance gain is larger when D, contains more noise and has larger domain shifts; (2) sparsity at transfer also beneficial after fine-tuning, and the improvement is more significant when the D, is more "data-scarce", (3) the optimal sparse levels for the two stages vary for different combinations of pretrain and transfer domains, highlighting the importance of choosing the correct pruning rounds for both stages.
the performance
Table A5: Test accuracy of fine-tuned ResNet- 18 on CUB-200 after pretrained on ImageNet, under different levels of sparsity at pretraining and sparsity at transfer.
Sparsity At Pretraining
Sparsity At Transfer
0.00%
20.00%
36.00%
48.80%
59.04%
67.23%
0.00%
74.16%
76.01 %
75.77%
75.87%
74.99%
74.35%
20.00%
74.15%
75.54%
75.82%
75.98%
74.73%
74.46%
36.00%
74.13%
75.08%
75.56%
75.73%
74.06%
73. 73.94%
48. 88,80%
73.84%
74.01%
74.56%
74.46%
72.37%
72. 12%16%
59.0 99.04%
73.61%
73.77%
73.61%
72.89%
70.66%
70.56%
Table A6: Test accuracy of fine-tuned ResNet-18 on CUB-200 after pretrained on ImageNet-C, under different levels of sparsity at pretraining and sparsity at transfer.
Sparsity At Pretraining
Sparsity At Transfer
0.00%
20.00%
36.00%
48.80%
59.04%
67.23%
0.00% 20.00%
71.59%
71.89%
73.44%
73.70%
73.63%
73.52%
71.83%
72.44%
73.97%
74.01%
73.52%
73.39%
36.00%
71.68%
72.80%
73.46%
73.47%
72.95%
72.70%
48.80%
71.13%
71.87%
72.14%
72.40%
71.87%
71.28%
59.04%
69.26%
70.31%
70.61%
70.73%
70.11%
69.38%
B.3 MORE ABLATIONS ON IMAGE DATA
Effects of sparsity at two stages. We conduct set of ablation experiments to study the effects of two sparse masks in the Bi-RPT formulation on ResNet-18 (pretrained by ImageNet-C, fine-tuned on CUB-200). We compare against three baselines: fixing m., fixing m,, and fixing both of them.
A17

---

Under review as a conference paper at ICLR 2023
Table A7: Test accuracy of fine-tuned ResNet-18 on CUB-200 (10-shot) after pretrained on Ima- geNet, under different levels of sparsity at pretraining and sparsity at transfer.
Sparsity At Pretraining
Sparsity At Transfer
0.00%
20.00%
36.00%
48.80%
59.04%
67.23
0.00%
38.66%
35.88%
38.30%
39.23%
40.73%
39.14%
20.00%
38.90%
36.56%
38.66%
40. 14%
40.78%
39.33%
36.00%
38.42%
36.31%
38.95%
40.14%
40.32%
38.97%
48.80%
38.13%
35.23%
37.80%
38.02%
38.37%
35.69%
59.04%
37.02%
33.21%
35.83%
35.54%
35.55%
32.78%
Table A8:
Test accuracy of fine-tuned ResNet-18 CUB-200 (10-shot) after pretrained ImageNet-C, under different levels of sparsity at pretraining and sparsity at transfer.
Sparsity At Pretraining
Sparsity At Transfer
0.00%
20.00%
36.00%
48.80%
59.04%
67.23%
0.00%
32.14%
34.29%
38.07%
36.
38.21% %
36.85%
20.00%
32.53%
35.99%
39.63%
37.92%
39.94%
37.66%
36.00%
32.52%
36.07%
38.99%
38.56%
39.07%
36.95%
48.80%
31.69%
34.85%
37.59%
36.45%
36.69%
34.79%
59.04%
30.64%
32.55%
34.79%
13.31%
33.72%
32.64%
The performance comparison shown in Table A9 stage yields the highest performance.
where we can see that learning masks both
Effects of y We conduct set of ablation experiments study the effects of diffferent y again on ResNet-18 (pretrained by ImageNet-C, fine-tuned on CUB-200), We vary y with in i.e., 1, x 10-4, and we present the results in Table A10 We show that x 10-4 yields the highest performance among all the choices.
Effects of learning rates, We conduct set of ablation experiments on ResNet-18 (pretrained by ImageNet-C, fine-tuned on CUB-200) to study the effects of different learning rate m. and m, The learning rates we study in this ablation experiments are (2.5, 3.0, 3.5, 4.0, 4.5, 5.0). We present the test accuracies in TableAll and we observe that Bi-RPT can stably outperform baselines (74.01%) within a wide range of Am
Table A9: Ablation study on dif- ferent sparse masks image data. "Fixed" means the value of h changed. We study the combina- tion of pretraining on ImageNet-C and transferring to Birds.
Table A10: Ablation study on the effects of different the image data. Test accuracy of fine-tuned ResNet-18 on CUB-200 after pre- trained on ImageNet-C is reported.
Table A11: Ablation study on the effects of different learning rate on m. and m. image data. Test accuracy fine-tuned ResNet-18 on CUB-200 after pretrained on ImageNet-C is reported.
Test Accuracy
Mask Type
Test Accuracy
Am
Test Accuracy
0.5x
72.32%
Fixed and
71.58%
Fixed ma
72.09%
1x10-1
76.29%
2.5
72.88%
Fixed m.
75.53%
2x10-4
65.42%
3.0
75. 75.73%
Ours (Bi-RPT)
76.29%
3x10
52.59%
3.5
76.29%
4.0
75.94%
4.5
73.69%
5.0
73.34%
B.4 VISUALIZATION
We visualize the sparsity pattern learned by Bi-RPT stages in Figure A4
A18

---

Under review as a conference paper at ICLR 2023
CUB-200, ImageNet
CUB-200, ImageNet-C
0.75
0.75
Sponsity
Spursity
0.50
15
20
20
Layer Index
Layer Index
CUB-200 (10-shot), ImageNet
CUB-200 (10-shot), ImageNot-C
0.88
0.8-
0.5
Spursity
Spursity
0.4
0.4
0.2
0.0
10
15
20
10
15
20
Layer Index
Layer Index
Combined
Figure A4: Layerwise sparsity learned by Bi-RPT on CUB-200 and Birds-S with ImageNet and ImageNet-C pretraining. We report the sparsity level of the two masks, as well as their combined sparsity (note that Bi-RPT allows for the two masks to partially overlap).
B.5 HEA DATA REPRESENTATIONS
The raw imputs for our ML predictor are the alloy's composition and the temperature where the experiment is conducted; therefore, they are 11-dimensional vectors. We map these vectors into 2D images following the pipeline shown in Figure A3 Given a formulation of an alloy, the periodic table representation (PTR) sets the percentage of each element into a specifie position according to its position in the periodic table; and the randomized periodic table representation (RPTR) sets the percentage of each element with pre-defined shuffled periodic table. In our experiments, we use the RPTR to map values in a more balanced way.
C ADDITIONAL EXPERIMENTS
C.1 UNCERTAINTY QUANTIFICATION
We provide additional analysis of uncertain quantification. We ensemble ten models trained with Bi-RPT and pretrain-and -nd--aanndttttrrreeeee-ee (PT) methods by averaging their predictions (Lakshminarayana) et al. 2017). and calculate the standard deviation of the predictions the uncertainty. The results after ensemble are shown in Table A12
show that ensemble of sparse models provides more reliable results compared to the pretrain- and-transfer baseline. Compared with the ensemble of dense models (PT), the ensemble of sparse models also exhibits strong correlation between the uncertainty and the prediction error.
D DATASET COMPARISON
We have provided a comparison between different relevant datasets in Table A13 We elaborate more on the differences:
1. Maresca & Curtin 2020) have only sparse data from the Mo-Nb-TT Ta- V-W element family.
2. Lee et al. 2201 a has released a database of the predicted yield strength of 10 million alloys family at 1300 K. Our dataset contains alloys
from the AICT-MM--Nb-TT---W-HHEE
A19

---

Under review as a conference paper at ICLR 2023
He
Li
Be
B
C
N
O
F
Ne
Na
Mg
A1
Si
S
C1
Ar
K
Ca
Se
Ti
v
Cr
Mn
Fe
Co
Ni
Cu
Zn
Ga
Ge
As
Se
Br
Kr
Rb
Sr
Y
Zr
Nb
Mo
Tc
Ru
Rh
Pd
Ag
Cd
In
Sn
Sb
Te
Xe
Cs
Ha
Lu
Hf
Ta
W
Re
Os
Ir
Pt
Au
Hg
'1
Pb
Bi
Po
At
Rn
Fr
Ra
Lr
La
Ce
Pr
Nd
Pm
Sm
Eu
Gd
Tb
Dy
Ho
Er
Tm
Yb
Ac
Th
Pa
U
Np
Am
Cm
Bk
Cr
Es
Fm
Md
No
Periodic Table Representation
0
0
0
0
0
0
0
O
O
T- (1473.15 200) 22000
0
O
O
O
O
O
O
Mog Nbg, Tag. Zr25
0
0
D2S
O
O
O
O
O
0
0
0
O
0
Random Periodic Table Representation
0
0
0
0
O
O
0
O
(1473.15 200)
O
O
O
Mog Nbg, Tag; Zr25
O
O
O
O
O
1.250
O
U
O
O
0
0
0
2.25
0
Figure A5: The pipeline for converting raw input into pseudoimage. The temperature is embed- ded as the value of the second channel.
family at temperatures from 300 K to 2500 K. Our simulation data are significantly larger (over billion samples). The whole simulation data will be available, while only 100K included for training the MI models in this study.
from
3. Borg et al. (2020) compiles experimental data from published material science articles since 2004. The dataset contains 630 samples with different crystal structures. Our experimen- tal dataset also compiles experimental data from published material science articles too, but have also sub-selected the data points using material science domain knowledge. Specifically, we only focus on alloys with BCC structures in contrast to Borg et al. (2000)
A20

---

Under review as a conference paper at ICLR 2023
Table A12: Uncertainty estimation calculated by ensembling independently trained models. We study two methods: iettrain-and-rrnneee (PT) and Bi-RPT. The results after ensemble are reported as PT-Ensemble and Bi-RPT-Ensenne respectively. The estimated uncertainty is reported in brack- ets.
Predicted Yield Stress (GPa)
Alloys
Temperature (K)
Experimental (GPa)
Bi-RPT
Bi-RPT-Ensemble
PT
PT-Ensemble
293.15
1.078
1.158 (0.083)
1.062
1.054 (0.01 I)
1.210
473.15
0.965
1.046 (0.087)
0.902
0.908 (0.015)
0.868
MoNbTaTi
673.15
0.746
0.850 (0.085)
0.731
0.740 (0.026)
0,685
873.15
0.508
0.674 (0. 103)
0.584
0.604 (0.021)
0.593
1273.15
0.425
0,482 (0.088)
0.488
0.501 (0.018)
0.539
298.15
1.268
1.268 (0.098)
1,068
1.062 (0.011)
1.399
873.15
0.677
0.798 (0. 102)
0.607
0.624 (0.022)
0.689
MoNNTATIW
1073.15
0.618
0.681 (0.111)
0.523
0.528 (0.013)
0.674
1273.15
0.536
0.567 (0.124)
0.486
0.496 (0.017)
0.620
296.15
1.527
1.392 (0.122)
1.132
1.142 (0.021)
1.515
873.15
0.861
0.864 (0.098)
0.685
0.698 (0.017)
0.973
HIMONBTTTTZZ
1073.15
0.762
0.747 (0. I0S)
0.612
0.624 (0.022)
0.791
1273.15
0,662
0.646 (0.134)
0.573
0.587 (0.022)
0.753
Table A13: Comparison between different datasets.
Dataset
Alloy Family
Number of data points
Temperature
Maresca & Curtin 2020
Mo-Nb-Ta-V.N
Sparse 10 million
N/A 1300 K N/A
Lee et 20Zla
AI-Cr-MoNNb-T WA-.N--T-------TTTTT W------i-Zr
Borg 2020
N/A
630 billion
X- (Ours)
AL-Cr-FeMo--N-TTE V-W-HI-i-Z
300 K 2500
A21


Reviews:
[{"review_id": "pfASZxU2rut", "reviewer": "Reviewer_THpP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aimed to discover high-entropy alloys with high yield strength, by first predicting the yield strength. This work presented a dataset called X-Yield for this task and proposed a bi-level optimization method called Bi-RPT as their corss-quality few shot transfer workflow.", "summary_of_the_review": "This paper brings a very interesting problem in material science to the machine learning community. However, given the technical issues and potential integrity problems, I cannot recommend acceptance at this point but encourage the authors to address those issues in the rebuttal so that it's possible for me to adjust the rating.", "strength_and_weaknesses": "This work is very well motivated. Discovering new materials is an important task in science and making sure a good predictor exists is a primary sub goal. \n\nTo make machine learning methods work on real world tasks is not trivial. The authors have done extensive analyses on the dataset and specialized the method that not only works on their task of interest, but also has the potential to be used for other transfer learning tasks that involves multi-fidelity. Preliminary results were shown on image data.\n\nThe main weakness that I see is that the authors only focused on the prediction accuracy, and mentioned nothing about the uncertainty predictions that would be critical to the subsequent experimental design task for new material discovery. I worry that when proceeding to the next step, the authors will have to redo their modeling because of this missing uncertainty prediction component.\n\nMoreover, since discovering new material with high yield strength is the task, I think it wouldn't be too much trouble for this paper to show some preliminary results on simulated new material discovery given their existing dataset. This will also avoid the issue on missing component in the modeling choice in the first place.\n\nAnother weakness is the lack of baselines and related work. Closely related works are not discussed, e.g. on multi-fidelity machine learning, domain adaptation and sim to real transfer. The followings are some I found with a quick search.\n\nhttps://www.sciencedirect.com/science/article/abs/pii/S0927025616306188\nhttps://www.sciencedirect.com/science/article/abs/pii/S0045782517307612\nCitations in https://en.wikipedia.org/wiki/Multifidelity_simulation\npapers mentioned in https://www.v7labs.com/blog/domain-adaptation-guide\nhttps://link.springer.com/chapter/10.1007/978-3-030-33950-0_25\nhttps://arxiv.org/pdf/2009.13303.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MblTOHcVC_", "reviewer": "Reviewer_iZzs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- The paper targets determining yield strength for high entropy alloys which has numerous practical applications\n- The first contribution of the paper is to organize a large-scale material science dataset termed X-Yield. The dataset is composed of  >100K synthetic data points and 240 experimentally measured real data points.\n- Since there is a domain gap between the real and synthetic data, the authors propose a technique based on sparsity regularization to learn from source while adapting the model to the target data. ", "summary_of_the_review": "The paper proposes a new benchmark dataset which might be useful for the material science community. The paper is well written but I have a few concerns regarding baselines and novelty of the dataset. My initial rating is 6. ", "strength_and_weaknesses": "Strengths:\n- I like the problem setup, the applications of the proposed dataset and the approach are well motivated. \n- The paper proposes a new benchmark dataset for enabling machine learning research for HEA yield strength prediction.\n- I like the connections that the paper draws to few shot learning in the ML for science domain. \n- The paper proposes a technique for few-shot + domain transfer which is based on sparsity regularization. \n- The approach is fairly simple and potentially applicable elsewhere. The authors do verify that it works well compared to some simple baselines on a image based few-shot transfer task. \n- The experimental setup is sound and the proposed approach brings about improvements for the task at hand\n\nWeaknesses:\n- Baselines : The problem setup falls in the ML research areas of few-shot learning/cross-domain few shot learning, (supervised) domain adaptation/transfer learning. For example, refer to [a,b] These areas have been well studied and are very active areas of research. It is not clear why this particular approach was chosen, and how the approach compares to other popular approaches in these areas. \n- Section 3: It would help to add a table comparing the kinds of datasets that exist to supplement the Section 3.\n- Section 3: I found the construction of the real part of the dataset (in Dataset Construction) seems misleading - Till that point I was under the impression that both real and synthetic parts were contributions, but according to that paragraph the real part was curated from an existing one. Please explain how your subset is different. In my opinion the contributions should be adjusted as well. \n- What happens if you use the weight masking strategy without a two-stage approach ? How does that compare to the variants presented in Table 1 ? \n- Question : Given that only 11 elements were used, why encode this as an image - which is mostly sparse? \n- Question : Are all configurations of the selected elements realistic. What happens if the model is given an unrealistic combination ? \n- Suggestion : Please move \"Data representation\" above \"Architectures and Baselines\". I was not expecting a CNN-based model and it was not clear to my why that was used till I got to Data Representations. \n- Question : Fig. 3 : Is the drop in relative performance for 1273K due to less data in this regime ?\n\n\n\n[a] Guo, Yunhui, et al. \"A broader study of cross-domain few-shot learning.\" ECCV 2020\n[b] Zhuang, Fuzhen, et al. \"A comprehensive survey on transfer learning.\" Proceedings of the IEEE\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "37Oza1qKRV", "reviewer": "Reviewer_NW7Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a large-scale material science benchmark with 240 experimental measurements and over 100K simulated high-entropy alloy yield strength annotations. To address the scarcity of experimental annotations and the quality gap in the imperfectly simulated data, the authors present a noise-robust feature learning regularizer at the pre-training stage, and as a data-efficient learning regularizer at the few-shot transfer stage. ", "summary_of_the_review": "This work is interesting. However, some techniques have been widely used in other applications, which limits the novelty.", "strength_and_weaknesses": "Strength:\n1. This work presents a large-scale material science benchmark.\n2. This work presents a two-stage method to address the scarcity of experimental annotations and the quality gap in the imperfectly simulated data.\n\n\nWeaknesses: \n1. The simulated data would have much noise because of the gap between real and theoretical scenarios. How did you address this problem? Although the authors give a sparsity solution, it lacks theoretical analysis to explain its effectiveness.\n2. How did you tune the accurate sparsity in the paper? From tables A5-6, one could find that some sparsity rates are so specific.\n3. It is better to discuss this work with noisy labels. Why did you utilize the techniques of learning with noisy labels to address the noise?\n4. The utilized techniques are not originally proposed by the authors. These techniques have been widely used in other applications, which limits the novelty.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject