# A machine learning-based classification approach for phase diagram prediction

Guillaume Deffrennes, Kei Terayama, Taichi Abe, Ryo Tamura

► To cite this version:

Guillaume Deffrennes, Kei Terayama, Taichi Abe, Ryo Tamura. A machine learning-based classification approach for phase diagram prediction. Materials & Design, 2022, 215, pp.110497. 10.1016/j.matdes.2022.110497. hal-04145874

HAL Id: hal-04145874

https://hal.science/hal-04145874

Submitted on 29 Jun 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

# A machine learning-based classification approach for phase diagram prediction

<figure><img src="image_3.png" /></figure>

Check for
updates

Guillaume Deffrennes<sup>a,*</sup>, Kei Terayama<sup>b</sup>, Taichi Abe<sup>c,d</sup>, Ryo Tamura<sup>a,d,e,f,*</sup>

## HIGHLIGHTS

* The number of coexisting phases across the 800 K section of the Al-Cu-Mg-Si-Zn quinary ternaries is predicted via machine learning.

* An average prediction accuracy of 84% is achieved.

* The predictions are useful for guiding experimental investigations of phase diagrams, even when using small datasets.

* Important machine learning features can be generated from CALPHAD extrapolations from binaries into higher-order systems.

## ARTICLE INFO

**Article history:**
Received 19 January 2022
Revised 25 February 2022
Accepted 26 February 2022
Available online 01 March 2022

**Keywords:**
Phase diagrams
Machine learning
Alloys
CALPHAD

## GRAPHICAL ABSTRACT

Phase diagram prediction
using machine learning

New descriptors

<figure><img src="image_4.png" /></figure>

Thermodynamic properties

CALPHAD extrapolations

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

Identification of
key experiments

## ABSTRACT

Knowledge of phase diagrams is essential for material design as it helps in understanding microstructure evolution during processing. The determination of phase diagrams is thus one of the central tasks in materials science. When exploring new materials for which the phase diagram is unknown, experimentalists often try to determine the key experiments that should be performed by referencing known phase diagrams of similar systems. To enhance this practical strategy, we attempted to estimate unknown phase diagrams based on known phase diagrams using a machine learning-based classification approach. As a proof of concept, we focused on predicting the number of coexisting phases across the 800 K isothermal section of each of the 10 ternaries of the Al-Cu-Mg-Si-Zn system from the other 9 sections. To increase the prediction accuracy, we introduced new descriptors generated from the thermodynamic properties of the elements and CALPHAD extrapolations from lower-order systems. Using the random forest method, the presence of single-, two-, and three-phase domains was predicted with an average accuracy of 84% across all 10 considered sections with a standard deviation of 11%. The proposed approach represents a promising tool for assisting the investigator in developing new materials and determining phase equilibria efficiently.

© 2022 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

<footnote><sup>a</sup>International Center for Materials Nanoarchitectonics (WPI-MANA), National Institute for Materials Science, 1-1 Namiki, Tsukuba, Ibaraki 305-0044, Japan</footnote>

<footnote><sup>b</sup>Graduate School of Medical Life Science, Yokohama City University, 1-7-29, Suehiro-cho, Tsurumi-ku, Kanagawa 230-0045, Japan</footnote>

<footnote><sup>c</sup>Research Center for Structural Materials, National Institute for Materials Science, 1-2-1 Sengen, Tsukuba, Ibaraki 305-0047, Japan</footnote>

<footnote><sup>d</sup>Research and Services Division of Materials Data and Integrated System, National Institute for Materials Science, 1-1 Namiki, Tsukuba, Ibaraki 305-0044, Japan</footnote>

<footnote><sup>e</sup>RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan</footnote>

<footnote><sup>f</sup>Graduate School of Frontier Sciences, The University of Tokyo, 5-1-5 Kashiwa-no-ha, Kashiwa, Chiba 277-8561, Japan</footnote>

<footnote>* Corresponding authors at: National Institute for Materials Science, 1-1 Namiki, Tsukuba, Ibaraki 305-0044, Japan.
E-mail addresses: DEFFRENNES.Guillaume@nims.go.jp (G. Deffrennes), TAMURA.Ryo@nims.go.jp (R. Tamura).</footnote>
# 1. Introduction

Phase diagrams are widely used as roadmaps to support the development of materials, from their synthesis to their applications. Detailed knowledge of phase equilibria and the underlying thermodynamics enables the establishment of the link between processing and microstructure, which is a crucial step in modern material design [1-4]. Thus, the development of new strategies based on machine learning (ML) to promote more efficient determination of phase diagrams is of significant interest. Recently, two main approaches have been found to be particularly promising.

On the one hand, the combination of high-throughput experimentation (HTE) and ML has attracted attention. HTE techniques usually involve the synthesis of thin-film libraries using combinatorial sputtering [5-7] or diffusion multiples [8-10]. ML approaches then enable the rapid analysis of large volumes of data from HTE, further automating the phase diagram determination [11,12]. On the other hand, ML predictions can help in reducing the number of experiments required to explore unknown material systems. There have already been several successful studies in which new compounds were discovered based on these predictions [13]. In addition, an uncertainty sampling-based active learning strategy for recommending the most informative experiments to be performed for efficient determination of the phase diagrams has been proposed [14-16] and demonstrated in practice [17].

In this study, we attempt to estimate unknown phase diagrams based on known phase diagrams using a machine learning-based classification approach. Compared with the prediction of a given material property, the classification problem is complicated because a variety of phases may be stable depending on the constituent elements. Two ingenuities were introduced to succeed in this task. (i) Compositional descriptors suitable for phase diagram prediction are proposed. These are generated from the thermodynamic properties of the elements and CALPHAD (CALculation of PHAse Diagrams) extrapolations. (ii) The categories are narrowed by focusing on the number of coexisting phases instead of the phases. Although this is a simplified representation of phase diagrams, it still contains essential information on phase equilibria, and the predictions can thus be relevant for planning experiments. An overview of the workflow of this study is presented in Fig. 1.

As a demonstration, we focus on the constitutive ternary phase diagrams of the Al-Cu-Mg-Si-Zn system. This practical case is selected because reliable thermodynamic descriptions are available for each of these 10 ternaries, which are of significant industrial importance for light alloys. More specifically, the prediction of their 800 K isothermal section is examined. At this temperature, the descriptions are generally supported by many measurements,

and the liquid phase begins to stabilize in the majority of systems. To predict ternary sections, the random forest [18] and support vector machine (SVM) [19] classifiers are adopted, which are implemented using Scikit-learn [20].

# 2. Methods

## 2.1. Target ternary phase diagrams

The 10 ternary subsystems of the Al-Cu-Mg-Si-Zn quinary are set as the target for the present investigation. For each phase diagram, the phase equilibria data are retrieved across the entire 800 K isothermal section with steps of 2 at.% using the high-throughput calculation function of the Pandat software [21]. The thermodynamic descriptions adopted in the calculations are as follows: Al-Cu-Mg [22], Al-Cu-Si [23], Al-Cu-Zn [24], Al-Si-Zn [25], Al-Mg-Si [26], Al-Mg-Zn [27], Cu-Mg-Si [28], Cu-Mg-Zn [29], Cu-Si-Zn [30], and Mg-Si-Zn [31]. As a result, 1326 datapoints per system are obtained. For each point, the number of coexisting phases (one, two, or three) obtained by the CALPHAD calculations is used as the label corresponding to the category in the classification problem. In total, the number of single-, two-, and three-phase domains in the dataset is 2263, 6063, and 4934, respectively. The distribution of the classes for each of the 10 isothermal sections is presented in Supplementary Note A (Table S.1).

## 2.2. Compositional descriptors

To predict the phase diagrams with a high accuracy, we use three types of descriptors called "Magpie," "Thermo," and "CALPHAD." These are generated based on the composition of each element. For the "CALPHAD" descriptors, CALPHAD assessments of lower-order systems are necessary, while the properties of the pure elements are used for "Magpie" and "Thermo."

### 2.2.1. "Magpie" descriptors

For each datapoint investigated, five features are first obtained from the atomic concentration of the elements (at.% Al, Cu, Mg, Si, and Zn). In addition, the mean and absolute deviation of the properties referred to as "Magpie" in Table 1 are calculated from their values for the pure elements and the composition using the Magpie library [32]. This 37-dimensional descriptor set is referred to as the "Magpie" descriptor set. It is noted that this type of descriptor is widely used to predict material properties [13,32].

### 2.2.2. "Thermo" descriptors

The "Magpie" descriptors do not include any thermodynamic properties. However, such information can be expected to be use-

<figure><img src="image_7.png" /><figcaption>Fig. 1. Schematic of the phase diagram prediction workflow.</figcaption></figure>
<table><caption>**Table 1**

Compositional descriptors for the phase diagram prediction.</caption><thead><tr><th>Property</th><th>Abbreviation</th><th>Statistics</th><th>Reference</th></tr></thead><tbody><tr><td>Atomic percent of Al</td><td>Al</td><td rowspan="5">Absolute value</td><td rowspan="5">Composition (included in the "Magpie" set)</td></tr><tr><td>Atomic percent of Cu</td><td>Cu</td></tr><tr><td>Atomic percent of Mg</td><td>Mg</td></tr><tr><td>Atomic percent of Si</td><td>Si</td></tr><tr><td>Atomic percent of Zn</td><td>Zn</td></tr><tr><td>Atomic number</td><td>AtomicNum</td><td rowspan="10">Mean (mean) and average deviation (avg_dev) of the property among elements in composition</td><td rowspan="10">"Magpie"</td></tr><tr><td>Mendelev number</td><td>Mendelev</td></tr><tr><td>Periodic table row</td><td>Row</td></tr><tr><td>Periodic table column</td><td>Column</td></tr><tr><td>Atomic weight</td><td>AtomicWeight</td></tr><tr><td>Covalent radius</td><td>CovalentRad</td></tr><tr><td>Electronegativity</td><td>Electroneg</td></tr><tr><td>Volume (at 0 K from DFT)</td><td>Volume</td></tr><tr><td>Melting point</td><td>Melting T</td></tr><tr><td>Number of valence electrons</td><td>NValence</td><td rowspan="6">in composition</td><td rowspan="6">"Thermo"</td></tr><tr><td>Filled s orbitals</td><td>NsValence</td></tr><tr><td>Filled p orbitals</td><td>NpValence</td></tr><tr><td>Filled d orbitals</td><td>NdValence</td></tr><tr><td>Unfilled valence orbitals</td><td>NUnfilled</td></tr><tr><td>Unfilled s orbitals</td><td>NsUnfilled</td></tr><tr><td>Unfilled p orbitals</td><td>NpUnfilled</td><td rowspan="2">in composition</td><td rowspan="2">"Thermo"</td></tr><tr><td>Heat capacity at 800 K</td><td>Cp</td></tr><tr><td>Enthalpy at 800 K (relative to the enthalpy of the element in its stable state at 298 K)</td><td>H</td><td rowspan="4">Absolute value</td><td rowspan="4">"CALPHAD"</td></tr><tr><td>Entropy at 800 K</td><td>S</td></tr><tr><td>Enthalpy of melting</td><td>Melting H</td></tr><tr><td>Entropy of melting</td><td>Melting S</td></tr><tr><td>Number of coexisting phases as extrapolated from binary descriptions</td><td>Extrapolated n_phase</td><td rowspan="3">Absolute value</td><td rowspan="3">"CALPHAD"</td></tr><tr><td>Gibbs energy as extrapolated from binary descriptions</td><td>Extrapolated Gibbs</td></tr><tr><td>Excess Gibbs energy in the liquid phase as extrapolated from binary descriptions</td><td>Extrapolated Ex G Liq</td></tr></tbody></table>

ful for phase diagram prediction. Thus, in the second descriptor set, 10 new related features are introduced. The five thermodynamic properties referred to as "Thermo" in Table 1 are prepared for the pure elements using the SGTE database [33] in its 5.0 version. Their mean and absolute deviation are then calculated at each composition point as for the "Magpie" case. We refer to these 10 descriptors as the "Thermo" descriptor set. A Python script, DesPD (Descriptors for Phase Diagrams), is available at https://github.com/GuillaumeDeffrennes/DesPD to easily obtain these descriptors at any temperature and composition.

### 2.2.3. "CALPHAD" descriptors

At constant temperature, pressure, and composition, thermodynamic equilibrium is characterized by a minimum of the Gibbs energy. Therefore, once this thermodynamic potential is known for all the phases of a system, its phase diagram can be accessed. Within the CALPHAD framework, the Gibbs energy of the phases is modeled based on a building-block approach. For instance, the Gibbs energy of the quinary Al-Cu-Mg-Si-Zn liquid is obtained from the description of the pure elements in the phase, their two-by-two interactions, and their relatively small ternary interactions. This implies that the thermodynamic description of a multi-component system can be approximated using its subsystems. Herein, we propose the use of the information extrapolated from lower-order systems, i.e., from the binaries in the present case, as descriptors. Specifically, we introduce three features: (i) the number of coexisting phases extrapolated from binary descriptions, (ii) the Gibbs energy extrapolated from binary descriptions, and (iii) the excess Gibbs energy in the liquid phase extrapolated from binary descriptions. These are referred to as the "CALPHAD" descriptor set (Table 1). To obtain this type of descriptor, CALPHAD assessments of lower-order systems (here, the binaries) are necessary. Furthermore, the first two features, i.e., "Extrapolated n_phase"

and "Extrapolated Gibbs," have to be calculated using a thermodynamic calculation software, such as those listed in [34]. This is not a requirement for "Extrapolated Ex G Liq" as this descriptor can be obtained via a direct calculation from the interaction parameters (Appendix).

For our demonstration target, the thermodynamic description of the constitutive binaries of the Al-Cu-Mg-Si-Zn system are gathered into a database: Al-Cu [24], Al-Mg [35], Al-Si [36] (based on [37]), Al-Zn [38], Cu-Mg [36] (based on [39]), Cu-Si [23], Cu-Zn [40], Mg-Si [41], Mg-Zn [27], and Si-Zn [25]. On this basis, for each of the 10 ternaries considered in this study, point calculations are performed every 2 at.% across the 800 K isothermal section that is obtained by extrapolation from the binaries. It is emphasized that, when a ternary phase diagram is extrapolated from its binary subsystems, any ternary compounds or solutions of a third element in a binary phase will be missing (these two cases will be simply referred to as ternary phases hereafter). However, a reasonable estimate of the Gibbs energy of the phases that already exist in the unaries (e.g., the liquid phase) can be obtained.

A Python script is available at https://github.com/GuillaumeDeffrennes/DesPD to easily obtain the excess thermodynamic properties in the liquid phase as extrapolated from the binaries.

## 2.3. Machine learning classification model

### 2.3.1. Random forest classifier

The random forest classifier (RFC) [18] is an ensemble learning algorithm in which a large number of decision trees are individually trained using random subsamples of the dataset. For an input sample, the probability of each class is calculated by averaging the votes of the trees. One advantage of using RFCs is that they provide a measure of the relative importance of each descriptor to the prediction model, thereby facilitating a straightforward feature selec-
<figure><img src="image_8.png" /><figcaption>**Fig. 2.** Schematic of the nested approach for optimizing the model and evaluating its performance. CV stands for cross-validation.</figcaption></figure>

tion. Besides, RFCs naturally provide probability estimates for each class. In this study, the RFC is implemented using the Scikit-learn library [20]. In the RFC, there are two important hyperparameters: the number of trees in the forest (n_estimators) and the fraction of samples randomly drawn from the dataset to train each tree (max Samples). Thus, these are tuned by following the procedure described in Section 2.3.2.

### 2.3.2. *Model optimization and performance evaluation*

The ML workflow used in the present study is schematized in Fig. 2 and detailed below.

To evaluate the performance of the models, a 10-fold cross-validation is used: each of the 10 ternaries of the Al-Cu-Mg-Si-Zn system is in turn considered unknown, and its 800 K isothermal section is predicted using the classification model trained by the 9 other ternaries. The optimization of each classification model is performed by non-random nine-fold cross-validation; that is, each of the nine known ternaries is in turn taken as the validation set, while the remaining eight others form the training set. During this stage, hyperparameters are tuned, and features are selected such that the prediction accuracy is maximized following a three-step process.

1. A preliminary hyperparameter tuning is performed using a grid search approach in which “n_estimators” is varied from 50 to 200 in steps of 50, and “max_sample” from 0.2 to 0.7 in steps of 0.1. All the other parameters are set to default values in Scikit-learn 1.0.2 [20].

2. Feature selection is performed while keeping the hyperparameters fixed at the values selected in the previous step. (i) Feature importance is calculated across the nine-fold cross-validations, and on this basis, the least important features are eliminated. (ii) Using this smaller feature set, the classification models are trained, and step (i) is repeated. Step (ii) is repeated over several iterations until the set of features that maximizes the prediction accuracy is obtained. For more information on the number of features eliminated at each iteration, the example provided in Supplementary Note B (Fig. S1(c)) may be referred to.

3. The final hyperparameter tuning is performed based on the optimal set of features using a grid search approach. “n_estimators” is varied from 100 to 400 in steps of 100, and “max_sample” is varied from 0.2 to 0.8 in steps of 0.1. All the other parameters are set to default values in Scikit-learn [20].

A classification model with a high predictive potential is thus obtained from the nine ternaries and applied to the prediction of the phase diagram that is considered unknown. A practical example

of the model optimization procedure can be referred to in Supplementary Note B.

## 3. Results

### 3.1. *Classification performance*

The average of prediction accuracies realized using RFCs across the 10 considered ternary sections is summarized in Fig. 3(a). It is calculated from the confusion matrices presented in Supplementary Note A (Tables S2-4). It is noted that this metric corresponds to the accuracy calculated using nested cross-validation [42], which is also called double cross-validation (Fig. 2). Here, we compare the results obtained using three different feature sets: “Magpie,” “Magpie + Thermo,” and “Magpie + Thermo + CALPHAD.” In addition, the results obtained when extrapolating the ternaries from their constitutive binaries using CALPHAD calculations are also presented for comparison. It is emphasized that, in this study, a given A-B-C ternary isothermal section comprises 1326 discretized points, 150 of which are located on the A-B, A-C, and B-C binary edges. When a section is predicted, the points located on its binary edges are included in the test set, but they are already observed during training as they are also part of other known similar systems. If only the points inside the predicted ternary section are considered when evaluating the performance, i.e., if the 150 points located on the binary edges are eliminated from the test set, the average accuracies of the RFCs decrease from 0.62, 0.64, and 0.84 to 0.57, 0.60, and 0.82, respectively, for the three descriptor sets considered.

Firstly, each of the investigated RFCs significantly outperformed the benchmark case, for which classes were attributed randomly, as presented in Fig. 3(a). This is an evidence that the prediction strategy used in this study is working as expected.

Although the prediction accuracy is slightly increased, the “Magpie + Thermo” model does not significantly outperform the model trained using “Magpie” features only. However, as is presented shortly, “Thermo” descriptors rank high in the calculated feature importance, which implies that these descriptors must be important for predicting phase diagrams.

It is also noted that the isothermal sections obtained using CALPHAD extrapolations from the binaries are on average more accurate than the ones predicted based on the “Magpie” and “Magpie + Thermo” descriptor sets. However, it is emphasized that CALPHAD extrapolations and RFC predictions are fundamentally different; each approach has its own strengths, as is discussed in Section 4. Moreover, these first two RFCs result in relatively accurate predictions based only on the composition and properties of the pure ele-
<figure><img src="image_9.png" /><figcaption>**Fig. 3.** Classification performance of each model compared with CALPHAD extrapolations. (a) Average of prediction accuracies for the 10 isothermal sections. (b) Accuracy for each individual case. (c) Average of F1 scores for each class and of macro F1 scores across the 10 sections. In (a) and (c), the error bars represent the 95% confidence interval calculated from the 10 individual cases.</figcaption></figure>

ments, whereas additional information is required to perform CALPHAD extrapolations.

The ternary isotherms are on average best predicted by the RFC trained using the "Magpie + Thermo + CALPHAD" descriptor set. A 20% increase in the accuracy is observed as compared to the two other investigated models, i.e., "Magpie" and "Magpie + Thermo." This significant improvement resulted mainly from the addition of only three features. Among them, the number of coexisting phases calculated by extrapolation from the binaries ("Extrapolated n_phase") can be considered to be an estimate of the target variable. Therefore, the last RFC can be considered as a simplified case wherein the error from the CALPHAD extrapolations must

be predicted rather than the number of coexisting phases. This is similar to the so-called crude estimation of property strategy used in ML [43,44]. It is emphasized that, as shown in Fig. 3(b), when the CALPHAD extrapolations from the binaries are accurate, as in the Al-Mg-Si case, the results of the "Magpie + Thermo + CALPHAD" RFC are equally good. However, when they are inaccurate, as in the Al-Cu-Mg case, this RFC still performs better than the other two RFCs. The only case wherein CALPHAD extrapolations are found to be significantly more accurate than the "Magpie + Thermo + CALPHAD" RFC is the Mg-Si-Zn system. For this ternary, the probabilities of having two- or three-phase domains are both calculated to be close to 50% in the central part of its 800 K section (Supplementary Note D, Fig. S13). As a result, the predicted section is less accurate, although a precise understanding of the phase equilibria is still obtained by observing the probability distribution of each label. It is noted that, in addition to the RFC, we also use the SVM classifier [19] for these prediction problems, and slightly lower performances are obtained (Supplementary Note C).

Fig. 3(c) summarizes the average of F1 scores for each class and of macro F1 scores across the 10 considered ternary sections. It should be noted that, for each of the three RFCs, the F1 score of the "three-phase domain" class is undefined for two systems out of the 10 systems examined, wherein this outcome is never predicted. Thus, these cases are omitted from the calculation of the mean F1 score, which possibly results in an optimistically biased performance evaluation of this class. As shown in Fig. 3(c), the prediction of three-phase domains is the most difficult task. For this class, we confirmed that the precision of the RFCs is generally relatively high, but their recall is low. This indicates that insufficient points are labeled as three-phase domains in the predicted phase diagrams. In contrast, the single-phase domains are best predicted. This is especially true when using the "Magpie + Thermo + CALPHAD" descriptor set, wherein an average F1 score of 0.85 is obtained. This is a better performance than those of both CALPHAD extrapolations and other RFCs. We confirmed that adoption as a feature of the excess Gibbs energy in the liquid phase, as extrapolated from the binaries, played an important role in achieving this outcome. It is interesting to discuss this positive result in the context of the prediction of single-phase solid solution for multiprincipal element alloys (MPEAs, also referred to as high-entropy alloys). In related studies, the enthalpy of mixing is often considered as a descriptor [45–50]. However, it is calculated based on Miedema's scheme with the use of enthalpy data for liquids [51], and it was noted [51] that this treatment generally results in relatively large errors. Furthermore, only the configurational entropy is usually considered. It has been highlighted [51,52] that this ideal behavior is more of an exception than the norm in MPEAs. Therefore, it is expected that an enhanced performance can be realized by considering the excess thermodynamic properties as calculated using the CALPHAD framework. It is expected that a reliable thermodynamic description of these complex multicomponent solutions can be obtained via extrapolation from binaries and ternaries, as suggested by a recent study on the enthalpy of mixing of the Co-Cr-Fe-Mn-Ni FCC phase [53]. It is emphasized that both the mixing enthalpy and entropy can be calculated in a spreadsheet directly from the composition and interaction parameters found in published thermodynamic databases. For materials informatics studies wherein a large compositional space is investigated, simply considering extrapolations from binaries may be an interesting compromise. Because this is a nontrivial task for those unfamiliar with the CALPHAD formalism, the calculation of the excess thermodynamic properties by extrapolation from the binaries is detailed in the appendix. Furthermore, a Python script is made available to obtain these easily (Section 2.2.3).
<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /><figcaption>**Fig. 4.** Overview of feature selection when using (a) "Magpie," (b) "Magpie + Thermo," and (c) "Magpie + Thermo + CALPHAD" descriptor sets. The average number of selection and average ranking are summarized.</figcaption></figure>

## 3.2. Important features

For the present classification task, we consider feature selection to be a critical part of the model optimization. Indeed, before and after this stage, which is described as step 2 in Section 2.2.2, a flat increase in accuracy of 8.4% is observed on average. In comparison, hyperparameter tuning performed during steps 1 and 3 only resulted in variations in accuracy of approximately 1%. After the feature selection, 11 features remained on average without any significant shift for the three descriptor sets.

Fig. 4 presents the selected features for each descriptor set. Only the features that are selected in at least two of the 10 cases are presented herein. On comparing the results obtained using the "Magpie" and "Magpie + Thermo" descriptor sets, it is found that the

selected features differ significantly, although the prediction accuracies are close (Fig. 2). Indeed, in the case of "Magpie + Thermo," the thermodynamic properties tend to be selected as important features. These "Thermo" descriptors are also actively selected with the use of the "Magpie + Thermo + CALPHAD" descriptor set. This implies that these thermodynamic descriptors are useful for predicting phase diagrams. For the RFC trained using the "Magpie + Thermo + CALPHAD" set, the number of coexisting phases calculated by extrapolation from the binaries is the most important feature. Another noteworthy feature is the excess Gibbs energy in the liquid phase as extrapolated from the binaries. Indeed, this descriptor is selected in all 10 cases and ranked third on average in terms of feature importance.

## 3.3. Use of predictions to guide experiments

These predictions provide insights into phase equilibria and can thus be useful for planning experiments. To illustrate this, the Cu-Mg-Zn ternary system is considered as an example. This system is rather complicated, notably owing to the formation of the ternary Laves C36 phase and the significant solubility of Zn in the Cu₂Mg (Laves C15) phase. As a result, the 800 K isothermal section presented in Fig. 5 (a) differs significantly from that extrapolated from the binaries presented in Fig. 5 (b). As discussed in Section 3.1, the RFC tends to underestimate the probability of having three-phase domains. As a result, this class is almost absent from the section predicted using the "Magpie + Thermo + CALPHAD" descriptor set presented in Fig. 5 (c). In contrast, the prediction of single-phase domains is significantly better.

Fig. 5 (d)–(f) present the predicted probabilities of having single-, two-, and three-phase domains, respectively. Although the section extrapolated from the binaries is characterized by a significantly extended single-phase liquid domain (Fig. 5(b)), this behavior is not predicted by the RFC, as shown in Fig. 5(d). Furthermore, from Fig. 5(e) and (f), it appears that the predicted probabilities for two- or three-phase domains are large in the central part of the phase diagram. This is consistent with the real isothermal section presented in Fig. 5 (a). A preliminary understanding of the phase equilibria can thus be obtained from the predictions via a subjective analysis. For example, if the objective is to find three-phase domains in the phase diagram, the candidate points can be selected using the probability presented in Fig. 5(f), i.e., the dark-red region is a promising area. These areas are generally found to be consistent with the three-phase domains in real isothermal sections.

If both the extrapolated section obtained via CALPHAD calculations and the predicted section are available, the differences between these provide useful information for guiding experiments in a more objective manner. In Fig. 6, the probability of having another class than the one obtained via CALPHAD extrapolations is displayed at each investigated point of the Cu-Mg-Zn section. The dark areas are where the extrapolations are likely to be incorrect according to the model, possibly owing to the formation of ternary phases. These areas are thus where experiments should be conducted first. This strategy resembles the uncertainty sampling approach for constructing phase diagrams efficiently proposed in [14–17].

## 4. Discussion

The results obtained for each of the 10 isothermal sections and three descriptor sets considered are summarized in Supplementary Note D. In general, predictions by RFCs are useful in cases wherein the phase diagram features an appreciable proportion of single-phase domains, as in the Cu-Mg-Zn case discussed in Section 3.2
<figure><img src="image_13.png" /><figcaption>**Fig. 5.** 800 K isothermal section of the Cu-Mg-Zn as (a) calculated based on the assessment from Ref. [29] (real section), (b) extrapolated from descriptions of the Cu-Mg [36], Cu-Zn [40], and Mg-Zn [27] binaries by CALPHAD calculations, and (c) predicted by RFC using the "Magpie + Thermo + CALPHAD" descriptor set. (d), (e), and (f) present probabilities predicted by RFC of having single-, two-, and three-phase domains, respectively.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>**Fig. 6.** Probabilities across the Cu-Mg-Zn 800 K isothermal section of having another class than that obtained via CALPHAD extrapolations as predicted by RFC using the "Magpie + Thermo + CALPHAD" descriptor set.</figcaption></figure>

as well as the Al-Cu-Mg and Al-Cu-Zn systems. In contrast, limited information is obtained in the cases wherein three-phase domains are largely predominant, such as in the case of the Al-Cu-Si and Cu-Mg-Si systems. Therefore, a better performance is expected at relatively high temperatures close to the liquidus, where solutions extend into the diagram, but some compounds remain stable. Furthermore, the best prediction results are obtained when using the "Magpie + Thermo + CALPHAD" descriptor set for the Al-Mg-Si, Al-Si-Zn, and Mg-Si-Zn systems. In all three cases characterized by the absence of ternary phases, the 800 K isothermal section is predicted not only with a high level of accuracy, but also with a high level of confidence.

It is emphasized that CALPHAD extrapolations and the present ML predictions are different in nature. In CALPHAD, phase diagrams are calculated from the thermodynamic description of their constitutive subsystems. The CALPHAD method is successful in extrapolating quaternary and higher-order phase diagrams from binaries and ternaries. However, extrapolations of ternaries from binaries are less consistent, and a binary cannot be reliably extrapolated from unaries. An important limitation of CALPHAD extrapolations is that a compound, or the solubility of an element in a phase, cannot be predicted if it does not already appear in the subsystems. In the present ML approach, phase diagrams are predicted by analogy with other systems. In contrast to CALPHAD extrapolations, ML can provide indications of the presence of new phases, as discussed in Section 3.3. Moreover, a measure of confidence in the predictions can be obtained from the class probability distribution.

In the present dataset, the classes are relatively well-balanced. However, for other systems and temperatures, class imbalance can become a problem, in which case undersampling and oversampling techniques may be important in achieving good results.

This study is focused on cases wherein the phase diagram of interest is completely unknown, except for its constitutive binaries. If partial data are available in the ternary phase diagram, they can be directly included in the training set. The strategies for guiding the experiments discussed herein can thus become part of an iterative process, such as black-box optimization [54,55], wherein the newly obtained data are used to refine the predictions before planning for the next step. It would be interesting to investigate the extent to which the performance of the model could be improved at each iteration.

We should emphasize that this study is our first attempt at predicting unknown phase diagrams from known phase diagrams. A certain degree of success is achieved, although only nine phase diagrams are used to train the models, and the prediction target is the number of coexisting phases. Future perspectives for improving the present approach include increasing the size of the dataset, introducing other appropriate labels for each phase, and accounting for temperature variations. To achieve this, the phase diagram
data for various systems must be collected by performing CALPHAD calculations. These potential research directions are being considered for future works.

## 5. Conclusions

Phase diagram prediction using ML was investigated by focusing on the ternary subsystems of the Al-Cu-Mg-Si-Zn quinary. Each of the 10 ternaries was considered to be unknown, and the number of coexisting phases across its 800 K isothermal section was predicted from the 9 others. Special emphasis was placed on introducing new descriptors for predicting the phase diagrams. It is emphasized that impactful features were obtained from CALPHAD extrapolations from the binary systems. To the best of our knowledge, this is a new strategy for generating descriptors, and it significantly improved the model performance in the present study. Using the most complete set of descriptors and a random forest algorithm, an average accuracy of 84% with a standard deviation of 11% was obtained across the 10 considered ternary sections. It should be noted that the accuracy is dependent on the system, temperature, and training set. It was demonstrated that the predictions can provide insights into phase equilibria in unknown materials systems, especially when combined with CALPHAD extrapolations. It is thus concluded that the proposed approach is promising for promoting more efficient determination of phase diagrams. We are currently working on the construction of a larger dataset to expand our approach to other elements.

## CRediT authorship contribution statement

**Guillaume Deffrennes:** Data Curation, Formal Analysis, Investigation, Methodology, Visualization, Writing – Original Draft Preparation. **Kei Terayama:** Methodology, Writing – Review & Editing. **Taichi Abe:** Data Curation, Writing – Review & Editing. **Ryo Tamura:** Conceptualization, Funding Acquisition, Methodology, Supervision, Writing – Original Draft Preparation.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgements

This study was supported by a project subsidized by the Core Research for Evolutional Science and Technology (CREST) program of the Japan Science and Technology Agency (JST) (Grant No. JPMJCR17J2). We thank Masanori Enoki for sharing his thermodynamic databases.

## Data availability

All thermodynamic databases used in the calculations are available in the NIMS Computational Phase Diagram Database (CPDDB) [56]. The complete dataset used in this study is shared as supplementary material.

## Appendix A. CALPHAD extrapolation of the excess thermodynamic properties of a substitutional solution from its binary subsystems

For a given A-B binary substitutional solution denoted by $\phi$, the molar excess Gibbs energy $^{ex}G$ is modeled within the CALPHAD

framework using a Redlich-Kister polynomial that includes interaction parameters:

$$^{ex}G^{\phi} = x_A x_B \sum_{\nu} (^{\nu}a_{A,B}^{\phi} + ^{\nu}b_{A,B}^{\phi} T)(x_A - x_B)^{\nu} \quad (1)$$

with $x_i$ the molar fraction of constituent $i$, and $\nu$ an integer. $^{\nu}a_{ij}^{\phi}$ and $^{\nu}b_{ij}^{\phi}$ form the interaction parameters of order $\nu$, which can be obtained from CALPHAD publications in tables under the notation $^{\nu}L_{ij}^{\phi}$. In thermodynamic databases in TDB format such as those available in NIMS CPDDB [56], binary interaction parameters are found after the keyword "PARAMETER $L(\phi, i; \nu)$".

For a substitutional solution of $n$ constituents ($n > 2$), the main contribution to the excess Gibbs energy is obtained by extrapolation from its binary subsystems. Various extrapolation methods are available, as discussed in [57,58]. The Muggianu formalism is most commonly used because the contribution from each binary can be directly calculated using Eq. (1), without the requirement for any weighting. Thus, the contribution to the excess thermodynamic properties from the binaries is obtained by the summation of Eq. (1) across all the cases as follows:

$$\text{bin.ex}G^{\phi} = \sum_{i} \sum_{j} x_i x_j \sum_{\nu} (^{\nu}a_{ij}^{\phi} + ^{\nu}b_{ij}^{\phi} T)(x_i - x_j)^{\nu} \quad (2)$$

$$\text{bin.ex}H^{\phi} = \sum_{i} \sum_{j} x_i x_j \sum_{\nu} ^{\nu}a_{ij}^{\phi}(x_i - x_j)^{\nu} \quad (3)$$

$$\text{bin.ex}S^{\phi} = - \sum_{i} \sum_{j} x_i x_j \sum_{\nu} ^{\nu}b_{ij}^{\phi}(x_i - x_j)^{\nu} \quad (4)$$

where $\text{bin.ex}H^{\phi}$ and $\text{bin.ex}S^{\phi}$ are the excess enthalpy and excess entropy as extrapolated from the binaries, i.e., on disregarding ternary and eventual higher-order interactions between the elements. The excess enthalpy directly corresponds to the enthalpy of mixing, whereas the configurational entropy must be added to the excess entropy to obtain the entropy of mixing.

## Appendix B. Supplementary material

Supplementary data to this article can be found online at https://doi.org/10.1016/j.matdes.2022.110497.

## References

[1] G.B. Olson, C.J. Kuehmann, Materials genomics: From CALPHAD to flight, Scr. Mater. 70 (2014) 25–30, https://doi.org/10.1016/j.scriptamat.2013.08.032.

[2] W. Xiong, G.B. Olson, Cybermaterials: materials by design and accelerated insertion of materials, NPJ Comput. Mater. 2 (2016) 15009, https://doi.org/10.1038/njpcomputats.2015.9.

[3] X. Wang, W. Xiong, Uncertainty quantification and composition optimization for alloy additive manufacturing through a CALPHAD-based ICME framework, NPJ Comput. Mater. 6 (2020) 188, https://doi.org/10.1038/s41524-020-00454-9.

[4] D. Khatamsaz, A. Molkeri, R. Couperthwaite, J. James, R. Arróyave, A. Srivastava, D. Allaire, Adaptive active subspace-based efficient multifidelity materials design, Mater. Des. 209 (2021) 110001, https://doi.org/10.1016/j.matdes.2021.110001.

[5] A. Ludwig, Discovery of new materials using combinatorial synthesis and high-throughput characterization of thin-film materials libraries combined with computational methods, NPJ Comput. Mater. 5 (2019) 70, https://doi.org/10.1038/s41524-019-0205-0.

[6] H. Xing, B. Zhao, Y. Wang, X. Zhang, Y. Ren, N. Yan, T. Gao, J. Li, L. Zhang, H. Wang, Rapid construction of Fe–Co–Ni composition-phase map by combinatorial materials chip approach, ACS Comb. Sci. 20 (3) (2018) 127–131, https://doi.org/10.1021/acscombsci.7b00171.

[7] Z. Zhao, Y. Jin, P. Shi, Y. Xue, B. Zhao, Y. Zhang, F. Huang, P. Bi, Q. Wang, An improved high-throughput data processing based on combinatorial materials chip approach for rapid construction of Fe–Cr–Ni composition-phase map, ACS Comb. Sci. 21 (2019) 833–842, https://doi.org/10.1021/acscombsci.9b00149.

[8] J.-C. Zhao, X. Zheng, D.G. Cahill, High-throughput diffusion multiples, Mater. Today 8 (10) (2005) 28–37, https://doi.org/10.1016/S1369-7021(05)71122-6.

[9] L.L. Zhu, H.Y. Qi, L. Jiang, Z.P. Jin, J.-C. Zhao, Experimental determination of the Ni–Cr–Ru phase diagram and thermodynamic reassessments of the Cr–Ru and
Ni-Cr-Ru systems, Intermetallics 64 (2015) 86-95, https://doi.org/10.1016/j.intermet.2015.04.011.

[10] Z. Wang, L. Zhang, W. Li, Z. Qin, Z. Wang, Z. Li, L. Tan, L. Zhu, F. Liu, H. Han, L. Jiang, High throughput experiment assisted discovery of new Ni-base superalloys, Scr. Mater. 178 (2020) 134-138, https://doi.org/10.1016/j scriptamat.2019.11.019.

[11] A.G. Kusne, T. Gao, A. Mehta, L. Ke, M.C. Nguyen, K.-M. Ho, V. Antropov, C.-Z. Wang, M.J. Kramer, C. Long, I. Takeuchi, On-the-fly machine-learning for high-throughput experiments: search for rare-earth-free permanent magnets, Sci. Rep. 4 (2015) 6367, https://doi.org/10.1038/srep06367.

[12] Y. Iwasaki, A.G. Kusne, I. Takeuchi, Comparison of dissimilarity measures for cluster analysis of X-ray diffraction data from combinatorial libraries, NPJ Comput. Mater. 3 (2017) 4, https://doi.org/10.1038/s41524-017-0006-2.

[13] J.E. Saal, A.O. Oliynyk, B. Meredig, Machine learning in materials discovery: confirmed predictions and their underlying approaches, Annu. Rev. Mater. Res. 50 (1) (2020) 49-69, https://doi.org/10.1146/annurev-matsci-090319-010954.

[14] K. Terayama, R. Tamura, Y. Nose, H. Hiramatsu, H. Hosono, Y. Okuno, K. Tsuda, Efficient construction method for phase diagrams using uncertainty sampling, Phys. Rev. Mater. 3 (2019), 033802, https://doi.org/10.1103/PhysRevMaterials. 3.033802.

[15] K. Terayama, K. Tsuda, R. Tamura, Efficient recommendation tool of materials by an executable file based on machine learning, Jpn. J. Appl. Phys. 58 (9) (2019) 098001, https://doi.org/10.7567/1347-4065/ab349b.

[16] K. Terayama, K. Han, R. Katube, I. Ohnuma, T. Abe, Y. Nose, R. Tamura, Acceleration of phase diagram construction by machine learning incorporating Gibbs' phase rule, Scr. Mater. 208 (2022) 114335, https://doi.org/10.1016/j. scriptamat.2021.114335.

[17] R. Katube, K. Terayama, R. Tamura, Y. Nose, Experimental establishment of phase diagrams guided by uncertainty sampling: an application to the deposition of Zn-Sn-P films by molecular beam epitaxy, ACS Mater. Lett. 2 (6) (2020) 571-575, https://doi.org/10.1021/acsmaterialslett.0c00104.

[18] L. Breiman, Random forests, Mach. Learn. 45 (2001) 5-32, https://doi.org/ 10.1023/A:1010933404324.

[19] B.E. Boser, I.M. Guyon, V.N. Vapnik, A training algorithm for optimal margin classifiers, in: Proceedings of the Fifth Annual Workshop on Computational Learning Theory - COLT '92, ACM Press, Pittsburgh, Pennsylvania, United States, 1992, pp. 144-152. https://doi.org/10.1145/130385.130401.

[20] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, É. Duchesnay, Scikit-learn: Machine learning in Python, J. Mach. Learn. Res. 12 (2011) 2825-2830.

[21] W. Cao, S.-L. Chen, F. Zhang, K. Wu, Y. Yang, Y.A. Chang, R. Schmid-Fetzer, W.A. Oates, PANDAT software with PanEngine, PanOptimizer and PanPrecipitation for multi-component phase diagram calculation and materials property simulation, Calphad 33 (2) (2009) 328-342, https://doi.org/10.1016/ j.calphad.2008.08.004.

[22] N. Hayashi, K. Nakashima, M. Enoki, H. Ohtani, Thermodynamic analysis of the Al-Cu-Mg ternary system, J. Jpn. Inst. Met. 83 (2019) 378-387, https://doi.org/ 10.2320/jinstmet.JAW201903.

[23] B. Hallstedt, J. Gröbner, M. Hampl, R. Schmid-Fetzer, Calorimetric measurements and assessment of the binary Cu-Si and ternary Al-Cu-Si phase diagrams, Calphad 53 (2016) 25-38, https://doi.org/10.1016/ j.calphad.2016.03.002.

[24] S.-M. Liang, R. Schmid-Fetzer, Thermodynamic assessment of the Al-Cu-Zn system, Part III: Al-Cu-Zn ternary system, Calphad 52 (2016) 21-37, https:// doi.org/10.1016/j.calphad.2015.11.001.

[25] M.H.G. Jacobs, P.J. Spencer, A critical thermodynamic evaluation of the systems Si-Zn and Al-Si-Zn, Calphad 20 (3) (1996) 307-320, https://doi.org/10.1016/ S0364-5916(96)00033-8.

[26] J. Lacaze, R. Valdes, CALPHAD-type Assessment of the Al-Mg-Si System, Monatsh. Chem. 136 (11) (2005) 1899-1907, https://doi.org/10.1007/s00706- 005-0385-9.

[27] N. Hayashi, K. Nakashima, M. Enoki, H. Ohtani, Thermodynamic analysis of the Al-Mg-Zn ternary system, J. Jpn. Inst. Met. 84 (2020) 141-150, https://doi.org/ 10.2320/jinstmet.JBW201905.

[28] J. Miettinen, G. Vassilev, Thermodynamic assessment of the Cu-Mg-Si system in its copper-rich region, Cryst. Res. Technol. 46 (11) (2011) 1122-1130, https://doi.org/10.1002/crat.201100324.

[29] L. Dreval, Y. Zeng, O. Ovbenko, Y. Du, S. Liu, B. Hu, H. Zhang, Thermodynamic description and simulation of solidification microstructures in the Cu-Mg-Zn system, J. Mater. Sci. 56 (17) (2021) 10614-10639, https://doi.org/10.1007/ s10853-021-05818-0.

[30] J. Miettinen, Thermodynamic description of the Cu-Si-Zn system in the copper-rich corner, Calphad 31 (4) (2007) 422-427, https://doi.org/10.1016/ j.calphad.2007.05.003.

[31] M. Wang, P. Wang, S. Liu, Y. Du, H. Xu, W. Zhang, Experimental investigation and thermodynamic modeling of the Mg-Si-Zn system, Calphad 35 (2) (2011) 183-190, https://doi.org/10.1016/j.calphad.2011.01.003.

[32] L. Ward, A. Agrawal, A. Choudhary, C. Wolverton, A general-purpose machine learning framework for predicting properties of inorganic materials, NPJ Comput. Mater. 2 (2016) 16028, https://doi.org/10.1038/ npjcompumats.2016.28.

[33] A.T. Dinsdale, SGTE data for pure elements, Calphad 15 (4) (1991) 317-425, https://doi.org/10.1016/0364-5916(91)90030-N.

[34] U.R. Kattner, The need for reliable data in computational thermodynamics, High Temp. High Press. 49 (2020) 31-47, https://doi.org/10.32908/hthp. v49.853.

[35] Y.u. Zhong, M. Yang, Z.-K. Liu, Contribution of first-principles energetics to Al-Mg thermodynamic modeling, Calphad 29 (4) (2005) 303-311, https://doi.org/ 10.1016/j.calphad.2005.08.004.

[36] I. Ansara, A.T. Dinsdale, M.H. Rand (Eds.), Definition of thermochemical and thermophysical properties to provide a database for the development of new light alloys: COST 507. Vol. 2: Thermochemical database for light metal alloys, Office for Official Publications of the European Communities, Luxembourg, 1998.

[37] J. Gröbner, H.L. Lukas, F. Aldinger, Thermodynamic calculation of the ternary system Al-Si-C, Calphad 20 (2) (1996) 247-254, https://doi.org/10.1016/ S0364-5916(96)00027-2.

[38] S. An Mey, Reevaluation of the Al-Zn System, Int. J. Mater. Res. 84 (1993) 451-455, https://doi.org/10.1515/ijmr-1993-840704.

[39] C.A. Coughanowr, I. Ansara, R. Luoma, M. Hämäläinen, H.L. Lukas, Assessment of the Cu-Mg system / Optimierung des systems Cu-Mg, Int. J. Mater. Res. 82 (1991) 574-581, https://doi.org/10.1515/ijmr-1991-820711.

[40] S.-M. Liang, H.-M. Hsiao, R. Schmid-Fetzer, Thermodynamic assessment of the Al-Cu-Zn system, part I: Cu-Zn binary system, Calphad 51 (2015) 224-232, https://doi.org/10.1016/j.calphad.2015.09.010.

[41] D. Kevorkov, R. Schmid-Fetzer, F. Zhang, Phase equilibria and thermodynamics of the Mg-Si-Li system and remodeling of the Mg-Si system, J. Ph. Equilib. Diffus. 25 (2) (2004) 140-151, https://doi.org/10.1007/s11669-004-0007-y.

[42] S. Varma, R. Simon, Bias in error estimation when using cross-validation for model selection, BMC Bioinform. 7 (2006) 91, https://doi.org/10.1186/1471- 2105-7-91.

[43] Y. Zhang, C. Ling, A strategy to apply machine learning to small datasets in materials science, NPJ Comput. Mater. 4 (2018) 25, https://doi.org/10.1038/ s41524-018-0081-z.

[44] J. Schmidt, M.R.G. Marques, S. Botti, M.A.L. Marques, Recent advances and applications of machine learning in solid-state materials science, NPJ Comput. Mater. 5 (2019) 83, https://doi.org/10.1038/s41524-019-0221-0.

[45] Y. Zhang, Y.J. Zhou, J.P. Lin, G.L. Chen, P.K. Liaw, Solid-solution phase formation rules for multi-component alloys, Adv. Eng. Mater. 10 (6) (2008) 534-538, https://doi.org/10.1002/adem.200700240.

[46] S. Guo, Phase selection rules for cast high entropy alloys: an overview, Mater. Sci. Technol. 31 (10) (2015) 1223-1230, https://doi.org/10.1179/ 1743284715Y.00000000018.

[47] L. Zhang, H. Chen, X. Tao, H. Cai, J. Liu, Y. Ouyang, Q. Peng, Y. Du, Machine learning reveals the importance of the formation enthalpy and atom-size difference in forming phases of high entropy alloys, Mater. Des. 193 (2020) 108835, https://doi.org/10.1016/j.matdes.2020.108835.

[48] Y. Zeng, M. Man, K. Bai, Y.-W. Zhang, Revealing high-fidelity phase selection rules for high entropy alloys: A combined CALPHAD and machine learning study, Mater. Des. 202 (2021) 109532, https://doi.org/10.1016/ j.matdes.2021.109532.

[49] S.Y. Lee, S. Byeon, H.S. Kim, H. Jin, S. Lee, Deep learning-based phase prediction of high-entropy alloys: Optimization, generation, and explanation, Mater. Des. 197 (2021) 109260, https://doi.org/10.1016/j.matdes.2020.109260.

[50] X. Huang, C. Jin, C. Zhang, H. U. Zhang, H. Fu, Machine learning assisted modelling and design of solid solution hardened high entropy alloys, Mater. Des. 211 (2021) 110177, https://doi.org/10.1016/j.matdes.2021.110177.

[51] R. Feng, M. Gao, C. Lee, M. Mathes, T. Zuo, S. Chen, J. Hawk, Y. Zhang, P. Liaw, Design of light-weight high-entropy alloys, Entropy, 18 (2016) 333, https:// doi.org/10.3390/e18090333.

[52] A. Takeuchi, Mixing entropy of exact equiatomic high-entropy alloys formed into a single phase, Mater. Trans. 61 (9) (2020) 1717-1726, https://doi.org/ 10.2320/matertrans.MT-M2020141.

[53] G. Bracq, J.-C. Crivello, M. Laurent-Brocq, J.-M. Joubert, I. Guillot, What is the enthalpy contribution to the stabilization of the Co-Cr-Fe-Mn-Ni faced-centered cubic solid solution?, J Ph. Equilib. Diffus. 42 (5) (2021) 561-570, https://doi.org/10.1007/s11669-021-00922-9.

[54] K. Terayama, M. Sumita, R. Tamura, K. Tsuda, Black-box optimization for automated discovery, Acc. Chem. Res. 54 (6) (2021) 1334-1346, https://doi. org/10.1021/acs.accounts.0c00713.

[55] R. Tamura, T. Osada, K. Minagawa, T. Kohata, M. Hirosawa, K. Tsuda, K. Kawagishi, Machine learning-driven optimization in powder manufacturing of Ni-Co based superalloy, Mater. Des. 198 (2021) 109290, https://doi.org/ 10.1016/j.matdes.2020.109290.

[56] T. Abe, K. Hashimoto, Y. Sawada, K. Hirose, Y. Goto, NIMS Computational Phase Diagram Database CPDDB, 2004, http://cpddb.nims.go.jp/cpddb/periodic.htm. https://doi.org/10.48505/nims.3060.

[57] P. Chartrand, A.D. Pelton, On the choice of "geometric" thermodynamic models, J. Ph. Equilib. 21 (2) (2000) 141-147, https://doi.org/10.1361/ 105497100770340192.

[58] M. Hillert, Empirical methods of predicting and representing thermodynamic properties of ternary solution phases, Calphad 4 (1) (1980) 1-12, https://doi. org/10.1016/0364-5916(80)90016-4.