Package: apex-flow
APEX: Alloy Property EXplorer using simulationsAPEX: Alloy Property EXplorer using simulations, is a component of theAI Squareproject that involves the restructuring of theDP-Genauto_testmodule to develop a versatile and extensible Python package for general alloy property testing. This package enables users to conveniently establish a wide range of property-test workflows by utilizing various computational approaches, including support for LAMMPS, VASP, and ABACUS.New Features Update (v1.0)Add calculation function ofphononspectrum (v1.1.0)Decouple property calculations into individual sub-workflow to facilitate the customization of complex property functionsSupport one-click parallel submission of multiple workflowsSupportrunstep in the single step test mode (Interaction method similar toauto_test)Allow user to adjust concurrency for task submission viagroup_sizeandpool_sizeAllow user to customizesuffixof property calculation directory so that multiple tests with identical property templates but different settings can be run within one workflowRefactor and optimize the command line interactionEnhance robustness across diverse use scenarios, especially for the local debug modeTable of ContentsAPEX: Alloy Property EXplorer using simulationsNew Features Update (v1.0)Table of Contents1. Overview2. Easy Install3. User Guide3.1. Before submission3.1.1. Global Setting3.1.2. Calculation Parameters3.1.2.1. EOS3.1.2.2. Elastic3.1.2.3. Surface3.1.2.4. Vacancy3.1.2.5. Interstitial3.1.2.6. Gamma Line3.1.2.7. Phonon Spectrum3.2. Command3.2.1. Workflow Submission3.2.2. Single-Step Test4. Quick Start4.1. In the Bohrium4.2. In a Local Argo Service4.3. In a Local Environment1. OverviewAPEX adopts the functionality of the second-generation alloy properties calculations and is developed utilizing thedflowframework. By integrating the benefits of cloud-native workflows, APEX streamlines the intricate procedure of automatically testing various configurations and properties. Owing to its cloud-native characteristic, APEX provides users with a more intuitive and user-friendly interaction, enhancing the overall user experience by eliminating concerns related to process control, task scheduling, observability, and disaster tolerance.The comprehensive architecture of APEX is demonstrated below:Figure 1. APEX schematic diagramAPEX consists of three types of pre-definedworkflowthat users can submit:relaxation,property, andjoint. Therelaxationandpropertysub-workflow comprise three sequentialsteps:Make,Run, andPost, while thejointworkflow essentially combines therelaxationandpropertyworkflows into a comprehensive workflow.Therelaxationprocess begins with the initialPOSCARsupplied by the user, which is used to generate crucial data such as the final relaxed structure and its corresponding energy, forces, and virial tensor. This equilibrium state information is essential for input into thepropertyworkflow, enabling further calculations of alloy properties. Upon completion, the final results are automatically retrieved and downloaded to the original working directory.In both therelaxationandpropertyworkflows, theMakestep prepares the corresponding computational tasks. These tasks are then transferred to theRunstep that is responsible for task dispatch, calculation monitoring, and retrieval of completed tasks (implemented through theDPDispatcherplugin). Upon completion of all tasks, thePoststep is initiated to collect data and obtain the desired property results.APEX currently offers computation methods for the following alloy properties:Equation of State (EOS)Elastic constantsSurface energyInterstitial formation energyVacancy formation energyGeneralized stacking fault energy (Gamma line)Moreover, APEX supports three types of calculators:LAMMPSfor molecular dynamics simulations, andVASPandABACUSfor first-principles calculations.2. Easy InstallEasy install bypipinstallapex-flowYou may also clone the package firstly bygitclonehttps://github.com/deepmodeling/APEX.gitthen install APEX bycdAPEX
pipinstall.3. User Guide3.1. Before submissionIn APEX,three vital elementsneed to be prepared before submission of a workflow:One global JSON filecontaining parameters to configuredflowand other global settings (default: "./global.json")Calculation JSON filecontaining parameters related to calculation (relaxation and property test)Work directoryconsists of necessary files indicated in above JSON files together with initial structures (default: "./")3.1.1. Global SettingThe instructions regarding global configuration,dflow, andDPDispatcherspecific settings must be stored in a JSON format file. The table below describes some crucial keywords, classified into three categories:Basic configKey wordsData structureDefaultDescriptionapex_image_nameStringzhuoy/apex_amd64Image for step other thanrun. One can build this Docker image via preparedDockerfilerun_image_nameStringNoneImage of calculator forrunstep. Use{calculator}_image_nameto indicate corresponding image for higher priority.run_commandStringNoneShell command forrunstep. Use{calculator}_run_commandto indicate corresponding command for higher priority.group_sizeInt1Number of tasks per parallel run group.pool_sizeInt1For multi tasks per parallel group, the pool size of multiprocessing pool to handle each task (1 for serial, -1 for infinity)upload_python_packageOptional[List]NoneExtra python packages needed to be used in the container.debug_pool_workersInt1Pool size of parallel tasks running in the debug modeDflow configKey wordsData structureDefaultDescriptiondflow_hostStringhttps://127.0.0.1:2746Url of dflow serverk8s_api_serverStringhttps://127.0.0.1:2746Url of kubernetes API serverdflow_configOptional[Dict]NoneSpecify more detailed dflow config in a nested dictionary with higher priority (Seedflow documentfor more detail).dflow_s3_configOptional[Dict]NoneSpecify dflow s3 repository config in a nested dictionary with higher priority (Seedflow documentfor more detail).Dispatcher config(One may refer toDPDispatcher’s documentationfor details of the following parameters)Key wordsData structureDefaultDescriptioncontext_typeStringNoneContext type to connect to the remote serverbatch_typeStringNoneSystem to dispatch taskslocal_rootString"./"Local root pathremote_rootStringNoneRemote root pathremote_hostStringNoneRemote root pathremote_usernameStringNoneRemote user nameremote_passwordStringNoneRemote user passwordportInt22Remote portmachineOptional[Dict]NoneCompletemachine settingdictionary defined in theDPDispatcherwith higher priorityresourcesOptional[Dict]NoneCompleteresources settingdictionary defined in theDPDispatcherwith higher prioritytaskOptional[Dict]NoneCompletetask settingdictionary defined in theDPDispatcherwith higher priorityBohrium(additonal dispatcher config to be specified when you want to quickly adopt the pre-built dflow service or scientific computing resources on theBohrium platform)Key wordsData structureDefaultDescriptionemailStringNoneEmail of your Bohrium accountphoneStringNonePhone number of your Bohrium accountpasswordStringNonePassword of your Bohrium accountprogram_idIntNoneProgram ID of your Bohrium accountscass_typeStringNoneNode type provided by BohriumPlease refer to theQuick Startsection for various instances of global JSON examples in different situations.3.1.2. Calculation ParametersThe method for indicating parameters in alloy property calculations is akin to the previousdpgen.autotestapproach. There arethreecategories of JSON files that determine the parameters to be passed to APEX, based on their contents.Categories calculation parameter files:TypeFile formatDictionary containedUsageRelaxationjsonstructures;interaction;RelaxationForrelaxationworflowPropertyjsonstructures;interaction;PropertiesForpropertyworflowJointjsonstructures;interaction;Relaxation;PropertiesForrelaxation,propertyandjointworflowsIt should be noted that files such as POSCAR, located within thestructuredirectory, or any other files specified within the JSON file should be defined as relative path to theworking directoryand prepared in advanced.Below are three examples (for detailed explanations of each parameter, please refer to theHands-on_auto-testdocumentation for further information):Relaxation parameter file{"structures":["confs/std-*"],"interaction":{"type":"deepmd","model":"frozen_model.pb","type_map":{"Mo":0}},"relaxation":{"cal_setting":{"etol":0,"ftol":1e-10,"maxiter":5000,"maximal":500000}}}Property parameter file{"structures":["confs/std-*"],"interaction":{"type":"deepmd","model":"frozen_model.pb","type_map":{"Mo":0}},"properties":[{"type":"eos","skip":false,"vol_start":0.6,"vol_end":1.4,"vol_step":0.1,"cal_setting":{"etol":0,"ftol":1e-10}},{"type":"elastic","skip":false,"norm_deform":1e-2,"shear_deform":1e-2,"cal_setting":{"etol":0,"ftol":1e-10}}]}Joint parameter file{"structures":["confs/std-*"],"interaction":{"type":"deepmd","model":"frozen_model.pb","type_map":{"Mo":0}},"relaxation":{"cal_setting":{"etol":0,"ftol":1e-10,"maxiter":5000,"maximal":500000}},"properties":[{"type":"eos","skip":false,"vol_start":0.6,"vol_end":1.4,"vol_step":0.1,"cal_setting":{"etol":0,"ftol":1e-10}},{"type":"elastic","skip":false,"norm_deform":1e-2,"shear_deform":1e-2,"cal_setting":{"etol":0,"ftol":1e-10}}]}3.1.2.1. EOSKey wordsData structureExampleDescriptionvol_startFloat0.9The starting volume related to the equilibriumstructurevol_endFloat1.1The maximum volume related to the equilibriumstructurevol_stepFloat0.01The volume increment related to the equilibriumstructure3.1.2.2. ElasticKey wordsData structureExampleDescriptionnorm_deformFloat1.1The biggest volume related to the equilibriumstructureshear_deformFloat0.01The volume increment related to the equilibriumstructure3.1.2.3. SurfaceKey wordsData structureExampleDescriptionmin_slab_sizeInt10Minimum size of slab thicknessmin_vacuum_sizeInt11Minimum size of vacuume widthpert_xzFloat0.01Perturbation through xz direction used tocompute surface energy, default = 0.01max_millerInt2The maximum miller index number of surface generated3.1.2.4. VacancyKey wordsData structureExampleDescriptionsupercellList[Int][3, 3, 3]The supercell to be constructed, default = [1,1,1]3.1.2.5. InterstitialKey wordsData structureExampleDescriptioninsert_eleList[String]["Al"]The element to be insertedsupercellList[Int][3, 3, 3]The supercell to be constructed, default =[1,1,1]conf_filtersDict"min_dist": 1.5Filter out the undesirable configuration3.1.2.6. Gamma LineFigure 2. Schematic diagram of Gamma line calculationThe Gamma line (generalized stacking fault energy) function of APEX calculates energy of a series slab structures of specific crystal plane, which displaced in the middle along a slip vector as illustrated inFigure 2. In APEX, the slab structrures are defined by a plane miller index and two orthogonal directions (primary and secondary) on the plane. Theslip vector is always along the primary directionswith slip length defined by user or default settings. Thus, by indicatingplane_millerand theslip_direction(AKA, primary direction), a slip system can be defined.For most common slip systems in respect to FCC, BCC and HCP crystal structures, slip direction, secondary direction and default fractional slip lengths are already documented and listed below (Users arestrongly advisedto follow those pre-defined slip system, or may need to double-check the generated slab structure, as unexpected results may occur especially for system like HCP):FCCPlane miller indexSlip directionSecondary directionDefault slip length$(001)$$[100]$$[010]$$a$$(110)$$[\bar{1}10]$$[001]$$\sqrt{2}a$$(111)$$[11\bar{2}]$$[\bar{1}10]$$\sqrt{6}a$$(111)$$[\bar{1}\bar{1}2]$$[1\bar{1}0]$$\sqrt{6}a$$(111)$$[\bar{1}10]$$[\bar{1}\bar{1}2]$$\sqrt{2}a$$(111)$$[1\bar{1}0]$$[11\bar{2}]$$\sqrt{2}a$BCCPlane miller indexSlip directionSecondary directionDefault slip length$(001)$$[100]$$[010]$$a$$(111)$$[\bar{1}10]$$[\bar{1}\bar{1}2]$$\frac{\sqrt{2}}{2}a$$(110)$$[\bar{1}11]$$[00\bar{1}]$$\frac{\sqrt{3}}{2}a$$(110)$$[1\bar{1}\bar{1}]$$[001]$$\frac{\sqrt{3}}{2}a$$(112)$$[11\bar{1}]$$[\bar{1}10]$$\frac{\sqrt{3}}{2}a$$(112)$$[\bar{1}\bar{1}1]$$[1\bar{1}0]$$\frac{\sqrt{3}}{2}a$$(123)$$[11\bar{1}]$$[\bar{2}10]$$\frac{\sqrt{3}}{2}a$$(123)$$[\bar{1}\bar{1}1]$$[2\bar{1}0]$$\frac{\sqrt{3}}{2}a$HCP (Bravais lattice)Plane miller indexSlip directionSecondary directionDefault slip length$(0001)$$[2\bar{1}\bar{1}0]$$[01\bar{1}0]$$a$$(0001)$$[1\bar{1}00]$$[01\bar{1}0]$$\sqrt{3}a$$(0001)$$[10\bar{1}0]$$[01\bar{1}0]$$\sqrt{3}a$$(01\bar{1}0)$$[\bar{2}110]$$[000\bar{1}]$$a$$(01\bar{1}0)$$[0001]$$[\bar{2}110]$$c$$(01\bar{1}0)$$[\bar{2}113]$$[000\bar{1}]$$\sqrt{a^2+c^2}$$(\bar{1}2\bar{1}0)$$[\bar{1}010]$$[000\bar{1}]$$\sqrt{3}a$$(\bar{1}2\bar{1}0)$$[0001]$$[\bar{1}010]$$c$$(01\bar{1}1)$$[\bar{2}110]$$[\bar{1}2\bar{1}\bar{3}]$$a$$(01\bar{1}1)$$[\bar{1}2\bar{1}\bar{3}]$$[2\bar{1}\bar{1}0]$$\sqrt{a^2+c^2}$$(01\bar{1}1)$$[0\bar{1}12]$$[\bar{1}2\bar{1}\bar{3}]$$\sqrt{3a^2+4c^2}$$(\bar{1}2\bar{1}2)$$[10\bar{1}0]$$[1\bar{2}13]$$\sqrt{3}a$$(\bar{1}2\bar{1}2)$$[1\bar{2}13]$$[\bar{1}010]$$\sqrt{a^2+c^2}$The parameters related to Gamma line calculation are listed below:Key wordsData structureDefaultDescriptionplane_millerSequence[Int]NoneMiller index of the target slabslip_directionSequence[Int]NoneMiller index of slip (primary) direction of the slabslip_lengthInt|Float; Sequence[Int|Float, Int|Float, Int|Float]Refer to specific slip system as the table shows above, or 1 if not indicatedSlip length along the primary direction with default unit set by user or default setting. As for format of[x, y, z], the length equals to $\sqrt{(xa)^2+(yb)^2+(zc)^2}$plane_shiftInt|Float0Shift of displacement plane with unit of lattice parameter$c$(positive for upwards). This allows creating slip plane within narrowly-spaced planes (seeref).n_stepsInt10Number of steps to displace slab along the slip vectorvacuum_sizeInt|Float0Thickness of vacuum layer added around the slab with unit of Angstromsupercell_sizeSequence[Int, Int, Int][1, 1, 5]Size of generated supper cell based on slab structureadd fixSequence[Str, Str, Str]["true","true","false"]Whether to add fix position constraint along x, y and z direction during calculationHere is an example:{"type":"gamma","skip":true,"plane_miller":[0,0,1],"slip_direction":[1,0,0],"hcp":{"plane_miller":[0,1,-1,1],"slip_direction":[-2,1,1,0],"slip_length":[1,0,1],"plane_shift":0.25},"supercell_size":[1,1,6],"vacuum_size":10,"add_fix":["true","true","false"],"n_steps":10}It should be noted that for various crystal structures,users can further define slip parameters within the respective nested dictionaries, which will be prioritized for adoption. In above example, the slip system configuration within the "hcp" dictionary will be utilized.3.1.2.7. Phonon SpectrumThis function incorporates part ofdflow-phononcodes into APEX to make it more complete. This workflow is realized viaPhonopy, and plusphonoLAMMPSfor LAMMPS calculation.IMPORTANT!!: it should be noticed that one will need thephonoLAMMPSpackage pre-installed within one'srun_imagefor properLAMMPScalculation of phonon spectrum.The parameters related toPhononcalculation are listed below:Key wordsData structureDefaultDescriptionprimitiveBoolFalseWhether to find primitive lattice structure for phonon calculationapproachString"linear"Specify phonon calculation method when usingVASP; Two options: 1. "linear" for theLinear Response Method, and 2. "displacement" for theFinite Displacement Methodsupercell_sizeSequence[Int][2, 2, 2]Size of supercell created for calculationMESHSequence[Int]NoneSpecify the dimensions of the grid in reciprocal space for which the phonon frequencies and eigenvectors are to be calculated. For example: [8, 8, 8]; Refer toPhonopy MESHPRIMITIVE_AXESStringNoneTo define the basis vectors of a primitive cell in terms of the basis vectors of a conventional cell for input cell transformation. For example: "0.0 0.5 0.5 0.5 0.0 0.5 0.5 0.5 0.0"; Refer toPhonopy PRIMITIVE_AXESBANDStringNoneIndicate band path in reciprocal space as format ofPhonopy BAND; For example: "0 0 0 1/2 0 1/2, 1/2 1/2 1 0 0 0 1/2 1/2 1/2"BAND_POINTSInt51Number of sampling points including the path endsBAND_CONNECTIONBoolTrueWith this option, band connections are estimated from eigenvectors and band structure is drawn considering band crossings. In sensitive cases, to obtain better band connections, it requires to increase number of points calculated in band segments by theBAND_POINTStag.When utilize theVASP, you havetwoprimary calculation methods at your disposal: theLinear Response Methodand theFinite Displacement Method.TheLinear Response Methodhas an edge over the Finite Displacement Method in that it eliminates the need for creating super-cells, thereby offering computational efficiency in certain cases. Additionally, this method is particularly well-suited for systems with anomalous phonon dispersion (like systems with Kohn anomalies), as it can precisely calculate the phonons at the specified points.On the other hand, theFinite Displacement Method's advantage lies in its versatility; it functions as an add-on compatible with any code, including those beyond the scope of density functional theory. The only requirement is that the external code can compute forces. For instance, ABACUS may lack an implementation of the Linear Response Method, but can effectively utilize the Finite Displacement Method implemented in phonon calculation.3.2. CommandAPEX currently supports two seperate run modes:workflow submission(running via dflow) andsingle-step test(running without dflow).3.2.1. Workflow SubmissionAPEX will execute a specific dflow workflow upon each invocation of the command in the format:apex submit [-h] [-c [CONFIG]] [-w WORK [WORK ...]] [-d] [-f {relax,props,joint}] parameter [parameter ...]. The type of workflow and calculation method will be automatically determined by APEX based on the parameter file provided by the user. Additionally, users can specify theworkflow type,configuration JSON file, andwork directorythrough an optional argument (Runapex submit -hfor help). Here is an example to submit ajointworkflow:apexsubmitparam_relax.jsonparam_props.json-c./global_bohrium.json-w'dp_demo_0?''eam_demo'if no config JSON and work directory is specified,./global.jsonand./will be passed as default values respectively.3.2.2. Single-Step TestAPEX also provides asingle-step test mode, which can runMakerunandPoststep individually under local enviornment.Please note that one needs to run command under the work directory in this mode.User can invoke them by format ofapex test [-h] [-m [MACHINE]] parameter {make_relax,run_relax,post_relax,make_props,run_props,post_props}(Runapex test -hfor help). Here is a example to do relaxation in this mode:Firstly, generate relaxation tasks byapextestparam_relax.jsonmake_relaxThen dispatch tasks byapextestparam_relax.jsonrun_relax-mmachine.jsonwheremachine.jsonis a JSON file to define dispatch method, containingmachine,resources,taskdictionaries andrun_commandas listed inDPDispatcher’s documentation. Here is an example to submit tasks to aSlurmmanaged remote HPC:{"run_command":"lmp -i in.lammps -v restart 0","machine":{"batch_type":"Slurm","context_type":"SSHContext","local_root":"./","remote_root":"/hpc/home/hku/zyl/Downloads/remote_tasks","remote_profile":{"hostname":"***.**.**.**","username":"USERNAME","password":"PASSWD","port":22,"timeout":10}},"resources":{"number_node":1,"cpu_per_node":4,"gpu_per_node":0,"queue_name":"apex_test","group_size":1,"module_list":["deepmd-kit/2.1.0/cpu_binary_release"],"custom_flags":["#SBATCH --partition=xlong","#SBATCH --ntasks=1","#SBATCH --mem=10G","#SBATCH --nodes=1","#SBATCH --time=1-00:00:00"]}}Finally, as all tasks are finished, post process byapextestparam_relax.jsonpost_relaxThe property test can follow similar approach.4. Quick StartWe present several case studies as introductory illustrations of APEX, tailored to distinct user scenarios. For our demonstration, we will utilize aLAMMPS_exampleto compute the Equation of State (EOS) and elastic constants of molybdenum in both Body-Centered Cubic (BCC) and Face-Centered Cubic (FCC) phases. To begin, we will examine the files prepared within the working directory for this specific case.lammps_demo
├── confs
│   ├── std-bcc
│   │   └── POSCAR
│   └── std-fcc
│       └── POSCAR
├── frozen_model.pb
├── global_bohrium.json
├── global_hpc.json
├── param_joint.json
├── param_props.json
└── param_relax.jsonThere are three types of parameter files and two types of global config files, as well as a force-field potential file of molybdenumfrozen_model.pb. Under the directory ofconfs, structure filePOSCARof both phases have been prepared respectively.4.1. In the BohriumThe most efficient method for submitting an APEX workflow is through the preconfigured execution environment of dflow on theBohrium platform. To do this, it may be necessary to create an account on Bohrium. Below is an example of a global.json file for this approach.{"dflow_host":"https://workflows.deepmodeling.com","k8s_api_server":"https://workflows.deepmodeling.com","batch_type":"Bohrium","context_type":"Bohrium","email":"YOUR_EMAIL","password":"YOUR_PASSWD","program_id":1234,"apex_image_name":"registry.dp.tech/dptech/prod-11045/apex-dependencies:0.0.3","lammps_image_name":"registry.dp.tech/dptech/prod-11045/deepmd-kit:deepmd-kit2.1.1_cuda11.6_gpu","lammps_run_command":"lmp -in in.lammps","scass_type":"c8_m31_1 * NVIDIA T4"}Then, one can submit a relaxation workflow via:apexsubmitparam_relax.json-cglobal_bohrium.jsonRemember to replace the values ofemail,passwordandprogram_idof your own before submit. As for image used, you can either built your own or use public images from Bohrium or pulling from the Docker Hub. Once the workflow is submitted, one can monitor it onhttps://workflows.deepmodeling.com.4.2. In a Local Argo ServiceAdditionally, a dflow environment can be installed on a local computer by executinginstallation scriptslocated in the dflow repository (User can also refer to thedflow service setup manualfor more details). For instance, to install on a Linux system without root access:bashinstall-linux-cn.shThis process will automatically configure the required local tools, including Docker, Minikube, and Argo service, with the default port set to127.0.0.1:2746. Consequently, one can modify theglobal_hpc.jsonfile to submit a workflow to this container without needing a Bohrium account. Here is an example:{"apex_image_name":"zhuoyli/apex_amd64","run_image_name":"zhuoyli/apex_amd64","run_command":"lmp -in in.lammps","batch_type":"Slurm","context_type":"SSHContext","local_root":"./","remote_root":"/hpc/home/zyl/Downloads/remote_tasks","remote_host":"123.12.12.12","remote_username":"USERNAME","remote_password":"PASSWD","resources":{"number_node":1,"cpu_per_node":4,"gpu_per_node":0,"queue_name":"apex_test","group_size":1,"module_list":["deepmd-kit/2.1.0/cpu_binary_release"],"custom_flags":["#SBATCH --partition=xlong","#SBATCH --ntasks=4","#SBATCH --mem=10G","#SBATCH --nodes=1","#SBATCH --time=1-00:00:00"]}}In this example, we attempt to distribute tasks to a remote node managed bySlurm. Users can replace the relevant parameters within themachinedictionary or specifyresourcesandtasksaccording toDPDispatcherrules.For the APEX image, it is publicly available onDocker Huband can be pulled automatically. Users may also choose to pull the image beforehand or create their own Docker image in the Minikube environment locally using aDockerfile(please refer toDocker's documentationfor building instructions) to expedite pod initialization.Then, one can submit a relaxation workflow via:apexsubmitparam_relax.json-cglobal_hpc.jsonUpon submission of the workflow, progress can be monitored athttps://127.0.0.1:2746.4.3. In a Local EnvironmentIf your local computer experiences difficulties connecting to the internet, APEX offers aworkflow local debug modethat allows the flow to operate in a basicPython3environment, independent of the Docker container. However, users willnotbe able to monitor the workflow through the Argo UI.To enable this feature, users can add an additional optional argument-dto the origin submission command, as demonstrated below:apexsubmit-dparam_relax.json-cglobal_hpc.jsonIn this approach, the user is not required to specify an image for executing APEX. Rather, APEX should be pre-installed in the defaultPython3environment to ensure proper functioning.