Package: dpdata
dpdatadpdatais a python package for manipulating data formats of software in computational science, including DeePMD-kit, VASP, LAMMPS, GROMACS, Gaussian.
dpdata only works with python 3.7 or above.InstallationOne can download the source code of dpdata bygitclonehttps://github.com/deepmodeling/dpdata.gitdpdatathen usepipto install the module from sourcecddpdata
pipinstall.dpdatacan also by install via pip without sourcepipinstalldpdataQuick startThis section gives some examples on how dpdata works. Firstly one needs to import the module in a python 3.x compatible code.importdpdataThe typicall workflow ofdpdataisLoad data from vasp or lammps or deepmd-kit data files.Manipulate dataDump data to in a desired formatLoad datad_poscar=dpdata.System("POSCAR",fmt="vasp/poscar")or let dpdata infer the format (vasp/poscar) of the file from the file name extensiond_poscar=dpdata.System("my.POSCAR")The number of atoms, atom types, coordinates are loaded from thePOSCARand stored to a dataSystemcalledd_poscar.
A dataSystem(a concept used bydeepmd-kit) contains frames that has the same number of atoms of the same type. The order of the atoms should be consistent among the frames in oneSystem.
It is noted thatPOSCARonly contains one frame.
If the multiple frames stored in, for example, aOUTCARis wanted,d_outcar=dpdata.LabeledSystem("OUTCAR")The labels provided in theOUTCAR, i.e. energies, forces and virials (if any), are loaded byLabeledSystem. It is noted that the forces of atoms are always assumed to exist.LabeledSystemis a derived class ofSystem.TheSystemorLabeledSystemcan be constructed from the following file formats with theformat keyin the table passed to argumentfmt:Softwareformatmulti frameslabeledclassformat keyvaspposcarFalseFalseSystem'vasp/poscar'vaspoutcarTrueTrueLabeledSystem'vasp/outcar'vaspxmlTrueTrueLabeledSystem'vasp/xml'lammpslmpFalseFalseSystem'lammps/lmp'lammpsdumpTrueFalseSystem'lammps/dump'deepmdrawTrueFalseSystem'deepmd/raw'deepmdnpyTrueFalseSystem'deepmd/npy'deepmdrawTrueTrueLabeledSystem'deepmd/raw'deepmdnpyTrueTrueLabeledSystem'deepmd/npy'deepmdnpyTrueTrueMultiSystems'deepmd/npy/mixed'deepmdnpyTrueFalseMultiSystems'deepmd/npy/mixed'gaussianlogFalseTrueLabeledSystem'gaussian/log'gaussianlogTrueTrueLabeledSystem'gaussian/md'siestaoutputFalseTrueLabeledSystem'siesta/output'siestaaimd_outputTrueTrueLabeledSystem'siesta/aimd_output'cp2k(deprecated in future)outputFalseTrueLabeledSystem'cp2k/output'cp2k(deprecated in future)aimd_outputTrueTrueLabeledSystem'cp2k/aimd_output'cp2k(plug-in)stdoutFalseTrueLabeledSystem'cp2kdata/e_f'cp2k(plug-in)stdoutTrueTrueLabeledSystem'cp2kdata/md'QElogFalseTrueLabeledSystem'qe/pw/scf'QElogTrueFalseSystem'qe/cp/traj'QElogTrueTrueLabeledSystem'qe/cp/traj'Fhi-aimsoutputTrueTrueLabeledSystem'fhi_aims/md'Fhi-aimsoutputFalseTrueLabeledSystem'fhi_aims/scf'quip/gapxyzTrueTrueMultiSystems'quip/gap/xyz'PWmatatom.configFalseFalseSystem'pwmat/atom.config'PWmatmovementTrueTrueLabeledSystem'pwmat/movement'PWmatOUT.MLMDTrueTrueLabeledSystem'pwmat/out.mlmd'AmbermultiTrueTrueLabeledSystem'amber/md'Amber/sqmsqm.outFalseFalseSystem'sqm/out'GromacsgroTrueFalseSystem'gromacs/gro'ABACUSSTRUFalseFalseSystem'abacus/stru'ABACUSSTRUFalseTrueLabeledSystem'abacus/scf'ABACUScifTrueTrueLabeledSystem'abacus/md'ABACUSSTRUTrueTrueLabeledSystem'abacus/relax'asestructureTrueTrueMultiSystems'ase/structure'DFTB+dftbplusFalseTrueLabeledSystem'dftbplus'The Classdpdata.MultiSystemscan read data  from a dir which may contains many files of different systems, or from single xyz file which contains different systems.Usedpdata.MultiSystems.from_dirto read from a  directory,dpdata.MultiSystemswill walk in the directory
Recursively  and  find all file with specific file_name. Supports all the file formats thatdpdata.LabeledSystemsupports.Usedpdata.MultiSystems.from_fileto read from single file. Single-file support is available for thequip/gap/xyzandase/structureformats.For example, forquip/gap xyzfiles, single .xyz file may contain many different configurations with different atom numbers and atom type.The following commands relating toClass dpdata.MultiSystemsmay be useful.# load dataxyz_multi_systems=dpdata.MultiSystems.from_file(file_name="tests/xyz/xyz_unittest.xyz",fmt="quip/gap/xyz")vasp_multi_systems=dpdata.MultiSystems.from_dir(dir_name="./mgal_outcar",file_name="OUTCAR",fmt="vasp/outcar")# use wildcardvasp_multi_systems=dpdata.MultiSystems.from_dir(dir_name="./mgal_outcar",file_name="*OUTCAR",fmt="vasp/outcar")# print the multi_system infomationprint(xyz_multi_systems)print(xyz_multi_systems.systems)# return a dictionaries# print the system infomationprint(xyz_multi_systems.systems["B1C9"].data)# dump a system's data to ./my_work_dir/B1C9_raw folderxyz_multi_systems.systems["B1C9"].to_deepmd_raw("./my_work_dir/B1C9_raw")# dump all systemsxyz_multi_systems.to_deepmd_raw("./my_deepmd_data/")You may also use the following code to parse muti-system:fromdpdataimportLabeledSystem,MultiSystemsfromglobimportglob"""process multi systems"""fs=glob("./*/OUTCAR")# remeber to change here !!!ms=MultiSystems()forfinfs:try:ls=LabeledSystem(f)except:print(f)iflen(ls)>0:ms.append(ls)ms.to_deepmd_raw("deepmd")ms.to_deepmd_npy("deepmd")Access dataThese properties stored inSystemandLabeledSystemcan be accessed by operator[]with the key of the property supplied, for examplecoords=d_outcar["coords"]Available properties are (nframe: number of frames in the system, natoms: total number of atoms in the system)keytypedimensionare labelsdescription'atom_names'list of strntypesFalseThe name of each atom type'atom_numbs'list of intntypesFalseThe number of atoms of each atom type'atom_types'np.ndarraynatomsFalseArray assigning type to each atom'cells'np.ndarraynframes x 3 x 3FalseThe cell tensor of each frame'coords'np.ndarraynframes x natoms x 3FalseThe atom coordinates'energies'np.ndarraynframesTrueThe frame energies'forces'np.ndarraynframes x natoms x 3TrueThe atom forces'virials'np.ndarraynframes x 3 x 3TrueThe virial tensor of each frameDump dataThe data stored inSystemorLabeledSystemcan be dumped in 'lammps/lmp' or 'vasp/poscar' format, for example:d_outcar.to("lammps/lmp","conf.lmp",frame_idx=0)The first frames ofd_outcarwill be dumped to 'conf.lmp'd_outcar.to("vasp/poscar","POSCAR",frame_idx=-1)The last frames ofd_outcarwill be dumped to 'POSCAR'.The data stored inLabeledSystemcan be dumped to deepmd-kit raw format, for exampled_outcar.to("deepmd/raw","dpmd_raw")Or a simpler command:dpdata.LabeledSystem("OUTCAR").to("deepmd/raw","dpmd_raw")Frame selection can be implemented bydpdata.LabeledSystem("OUTCAR").sub_system([0,-1]).to("deepmd/raw","dpmd_raw")by which only the first and last frames are dumped todpmd_raw.replicatedpdata will create a super cell of the current atom configuration.dpdata.System("./POSCAR").replicate((1,2,3,))tuple(1,2,3) means don't copy atom configuration in x direction, make 2 copys in y direction, make 3 copys in z direction.perturbBy the following example, each frame of the original system (dpdata.System('./POSCAR')) is perturbed to generate three new frames. For each frame, the cell is perturbed by 5% and the atom positions are perturbed by 0.6 Angstrom.atom_pert_styleindicates that the perturbation to the atom positions is subject to normal distribution. Other available options toatom_pert_styleareuniform(uniform in a ball), andconst(uniform on a sphere).perturbed_system=dpdata.System("./POSCAR").perturb(pert_num=3,cell_pert_fraction=0.05,atom_pert_distance=0.6,atom_pert_style="normal",)print(perturbed_system.data)replaceBy the following example, Random 8 Hf atoms in the system will be replaced by Zr atoms with the atom postion unchanged.s=dpdata.System("tests/poscars/POSCAR.P42nmc",fmt="vasp/poscar")s.replace("Hf","Zr",8)s.to_vasp_poscar("POSCAR.P42nmc.replace")BondOrderSystemA new classBondOrderSystemwhich inherits from classSystemis introduced in dpdata. This new class contains information of chemical bonds and formal charges (stored inBondOrderSystem.data['bonds'],BondOrderSystem.data['formal_charges']). Now BondOrderSystem can only read from .mol/.sdf formats, because of its dependency on rdkit (which means rdkit must be installed if you want to use this function). Other formats, such as pdb, must be converted to .mol/.sdf format (maybe with software like open babel).importdpdatasystem_1=dpdata.BondOrderSystem("tests/bond_order/CH3OH.mol",fmt="mol")# read from .mol filesystem_2=dpdata.BondOrderSystem("tests/bond_order/methane.sdf",fmt="sdf")# read from .sdf fileIn sdf file, all molecules must be of the same topology (i.e. conformers of the same molecular configuration).BondOrderSystemalso supports initialize from ardkit.Chem.rdchem.Molobject directly.fromrdkitimportChemfromrdkit.ChemimportAllChemimportdpdatamol=Chem.MolFromSmiles("CC")mol=Chem.AddHs(mol)AllChem.EmbedMultipleConfs(mol,10)system=dpdata.BondOrderSystem(rdkit_mol=mol)Bond Order AssignmentTheBondOrderSystemimplements a more robust sanitize procedure for rdkit Mol, as defined indpdata.rdkit.santizie.Sanitizer. This class defines 3 level of sanitization process by: low, medium and high. (default is medium).low: userdkit.Chem.SanitizeMol()function to sanitize molecule.medium: before using rdkit, the programm will first assign formal charge of each atom to avoid inappropriate valence exceptions. However, this mode requires the rightness of the bond order information in the given molecule.high: the program will try to fix inappropriate bond orders in aromatic hetreocycles, phosphate, sulfate, carboxyl, nitro, nitrine, guanidine groups. If this procedure fails to sanitize the given molecule, the program will then try to callobabelto pre-process the mol and repeat the sanitization procedure.That is to say, if you wan't to use this level of sanitization, please ensureobabelis installed in the environment.According to our test, our sanitization procedure can successfully read 4852 small molecules in the PDBBind-refined-set. It is necessary to point out that the in the molecule file (mol/sdf), the number of explicit hydrogens has to be correct. Thus, we recommend to useobabel xxx -O xxx -hto pre-process the file. The reason why we do not implement this hydrogen-adding procedure in dpdata is that we can not ensure its correctness.importdpdataforsdf_fileinglob.glob("bond_order/refined-set-ligands/obabel/*sdf"):syst=dpdata.BondOrderSystem(sdf_file,sanitize_level="high",verbose=False)Formal Charge AssignmentBondOrderSystem implement a method to assign formal charge for each atom based on the 8-electron rule (see below). Note that it only supports common elements in bio-system: B,C,N,O,P,S,Asimportdpdatasyst=dpdata.BondOrderSystem("tests/bond_order/CH3NH3+.mol",fmt="mol")print(syst.get_formal_charges())# return the formal charge on each atomprint(syst.get_charge())# return the total charge of the systemIf a valence of 3 is detected on carbon, the formal charge will be assigned to -1. Because for most cases (in alkynyl anion, isonitrile, cyclopentadienyl anion), the formal charge on 3-valence carbon is -1, and this is also consisent with the 8-electron rule.Mixed Type FormatThe formatdeepmd/npy/mixedis the mixed type numpy format for DeePMD-kit, and can be loaded or dumped through classdpdata.MultiSystems.Under this format, systems with the same number of atoms but different formula can be put together
for a larger system, especially when the frame numbers in systems are sparse.This also helps to mixture the type information together for model training with type embedding network.Here are examples usingdeepmd/npy/mixedformat:Dump a MultiSystems into a mixed type numpy directory:importdpdatadpdata.MultiSystems(*systems).to_deepmd_npy_mixed("mixed_dir")Load a mixed type data into a MultiSystems:importdpdatadpdata.MultiSystems().load_systems_from_file("mixed_dir",fmt="deepmd/npy/mixed")PluginsOne can followa simple exampleto add their own format by creating and installing plugins. It's critical to add theFormatclass toentry_points['dpdata.plugins']inpyproject.toml:[project.entry-points.'dpdata.plugins']random="dpdata_random:RandomFormat"