# Presentation: 484239

## MSCF for Dummies 2004                        			HP Itanium-2, Linux and Everything (Don’t panic!)

- MSCF Visualization and  User Services Group

## Outline

- Welcome and Introduction
- Session 1(9:00 am PST): General Machine Usage
- Getting access and login
- MPP2 overview
- Running jobs
- Allocations and accounting
- Useful commands and tips
- Support
- Available software
- Break
- Session 2 (10:45 am PST): Software development
- Compiling
- Debugging and profiling

## Molecular Science Computing Facility

## MSCF Science Drivers Increased Capability

**Subsurface  Reactive Transport**

**Biology**

**Nanoscience/Catalysis**

**New Computational Methods**

**Atmospheric Chemistry**

**Accurate Methods**

**Actinide Chemistry**

**Geobiochemistry**

## The MSCF Focus

- MSCF Computational Grand Challenge Projects
- >500Khrs/yr for 3 yrs for multi-institution research teams
- External scientific peer review
- “EMSL” Computational Science Research Pilot Projects
- <75Khrs for 1 yr
- Internal scientific peer review
- Strategic methods and software development in applications (collaborative problem solving environments, chemistry, biology, computer science, applied math, visualization, and systems utilities/libraries)
- Reviews by PNNL, SC/OBER, & MSCF Advisory Committee
- EMSL scientific data/model repository for computations and experiment

## Breadth of MSCF Scientific Impact233 publications in the past two years

- McCurdy, P, W Hess, and S Xantheas. Nitric acid-water complexes: theoretical calculations and comparison to experiment. Journal of Physical Chemistry A, 106(33), 7628-7635 (2002).
- Mcrae, R, G Schenter, B Garrett, Z Svetlicic, and D Truhlar. Variational transition state theory evaluation of the rate constant for proton transfer in a polar solvent. Journal of Chemical Physics, 115, 8460-8480 (2001).
- Merkx, M, DA Kopp, MH Sazinsky, et al. Dioxygen activation and methane hydroxylation by soluble methane monooxygenase: A tale of two irons and three proteins. Angewandte Chemie-International Edition, 2001. 40(15): p. 2782-2807
- Mielke, S, B Garrett, and K Peterson. A hierarchical family of global analytic Born-Oppenheimer potential energy surfaces for the H + H2 reaction ranging in quality from double-zeta to the complete basis set limit. Journal of Chemical Physics, 116(10), 4142-4161 (2002).
- Miller, J, W Wilson, D Lynch, K Wei, A Kurtulus, and H Trease. Low-LET microbeam dosimetry. Radiation Research, 158, 377-378 (2002).
- Miller, J, W Wilson, D Lynch, M Resat, and H Trease. Computational dosimetry for electron micorbeams: Monte Carlo track simulation combined with confocal microscopy. Radiation Research, 156, 438-439 (2001).
- Nakajima, T, T Yanai, and K Hirao. Relativistic electronic structure theory. Journal Of Computational Chemisty, 23, 847-860 (2002).
- Nikolova, RD, GN Vayssilov, N Rodios, and A Bojilova. Regio- and stereoselective [2+2] photodimerization of 3-substituted 2-alkoxy-2-oxo-2H-1,2-benzoxaphosphorines. Molecules, 7, 420-432 (2002).
- Niu S, X Wang, JA Nichols, L Wang, and Tn Ichiye.  2003.  “Combined Quantum Chemistry and Photoelectron Spectroscopy Study of the Electronic Structure and Reduction Potentials of Rubredoxin Redox Site Analogues.”  Journal of Physical Chemistry 107(16):2898-2907.
- Niu, S and NB Hall. Modeling the active sites in metalloenzymes 5. The heterolytic bond cleavage of H2 in the [NiFe] hydrogenase of Desulfovibrio gigas by a nucleophilic addition mechanism. Inorganic Chemistry, 40(24), 6201-6203 (2001).
- Orth RJ, JW Graybeal, DL Stiles, and JA Franz.  2003.  “Profitable to the Last Drop:  New Process Captures Valuable Components from Corn Fiber.”  Resource:  Engineering & Technology for a Sustainable World 10(3):5-6.
- Ouyang, M, JL Huang, CL Cheung, CM Lieber. Energy gaps in "metallic" single-walled carbon nanotubes. Science, 292(5517), 702-705 (2001).
- Panek, J, Z Latajka, and J Lundell. DFT calculations of HRgX (Rg = rare gas; X = halogen) molecules. Phys. Chem. Chem. Phys., 4, 2504-2510 (2002).
- Park, B, H Li, and L Corrales. Molecular dynamics simulation of La2O3-Na2O-SiO2 glasses: I. the structural role of La3+ cations. Journal of Non-crystalline Solids, 297(2-3), 220-238 (2002).
- Park, B, W Weber, and L Corrales. Molecular dynamics simulation study of threshold displacements and defect formation in zircon. Physical Review. B, Condensed Matter, 64(17), 174108, 1-16 (2001).
- Parthiban, S, G de Oliveira, and JML Martin. Benchmark ab initio energy profiles for the gas-phase SN2 reactions Y-+CH3X → CH3Y+X- (X,Y = F,Cl,Br). Validation of hybrid DFT methods. Journal of Physical Chemistry A, 105(5), 895-904 (2001).
- Paukstis, S, J Gole, D Dixon, and K Peterson. The ionization potential of Si2N and Si2O. Journal of Physical Chemistry A, 106(36), 8435-8411 (2002).
- Perkyns, JS, K Dyer, and BM Pettitt. Computationally useful bridge diagram series. II. Diagrams in H-bonds. Journal of Chemical Physics, 116, 9404-9412 (2002).
- Perrine K, and D Jones.  2003.  “Interactive Imaging Science on Parallel Computers:  Getting Immediate Results.”  In 17th International Parallel & Distributed Processing Symposium, Nice, France, IEEE Computer Society Press, Los Alamitos, California.
- Perrine, K, D Jones, P Hochschild, and R Swetz. 2002. “Interactive parallel visualization framework for distributed data.” In Proceedings VDA2002, SPIE Conference on Visualization and Data Analysis, San Jose, CA
- Peterson, K and M Gutowski. Electron binding energies of dipole-bound anions at the coupled cluster level with single, double, and triple excitations: HCN- and HNC-. Journal of Chemical Physics, 116(8), 3297-3299 (2002).
- Peterson, K and T Dunning. Accurate correlation consistent basis sets for molecular core-valence correlation effects: The second row atoms Al-Ar and the first row atoms B-Ne revisited. Journal of Chemical Physics, 117(23), 10548-10560 (2002).
- Petrie, G, C Dippold, G Fann, E Jurrus, B Moon, K Perrine, and D Jones. 2002. “Distributed computing approach for remote sensing data.” In Interface 2002, 34th Symposium on the Interface: Computing Science and Statistics, The Interface Foundation of North America, Montreal, Canada.
- Pruess, K, S Yabusaki, C Steefel, and P Lichtner. Fluid flow, heat transfer, and solute transport at nuclear waste storage tanks in the Hanford vadose zone. Abstracts of Papers of the American Chemical Society, GEOC Part 1, 222, 63 (2002).
- Re, S and K Morokuma. ONIOM study of chemical reactions in microsolvation clusters: (H2O)nCH3Cl+OH-(H2O)m (n+m = 1 and 2). Journal of Physical Chemistry A, 105(30), 7185-7197 (2001).
- Reho, J, J Higgins, M Nooijen, K Lehmann, G Scoles, and M Gutowski. Photoinduced non-adiabatic dynamics in quartet Na3 and K3 formed using helium nanodroplet isolation. Journal of Chemical Physics, 115(22), 10265-10274 (2001).
- Resat, H, H Wiley, and D Dixon. Probability-weighted dynamic Monte Carlo method for reaction kinetics simulations. Journal of Physical Chemistry B, 105(44), 11026-11034 (2001).
- Rosso KM, D Smith, and M Dupuis.  2003.  “An Ab Initio Model of Electron Transport in Hematite (-Fe2O3) Basal Planes.”  Journal of Chemical Physics 118(14):6455-6466.
- Rosso, KM and JR Rustad. Structures and energies of AlOOH and FeOOH polymorphs from plane wave pseudopotential calculations. American Mineralogist, 86, 312-317 (2001).
- Ruscic, B, A Wagner, L Harding, R Asher, D Feller, D Dixon, K Peterson, Y Song, X Qian, C Ng, J Liu, and C Wenwu. On the enthalpy of formation of hydroxyl radical and gas-phase bond dissociation energies of water and hydroxyl. Journal of Physical Chemistry A, 106(11), 2727-2747 (2002).
- Rustad, JR. Molecular models of surface relaxation, hydroxylation, and surface charging at oxide-water interfaces. Reviews in Mineralogy & Geochemistry, 42, 169-197 (2001).
- Salcedo, R, LMR Martinez, and JM Martinez-Magadan. Theoretical study of high-valent vanadium oxo-porphyrins as a dopant of crude oil. Journal of Molecular Structure-Theochem, 542, 115-121 (2001).
- Sastre, G, V Fornes, and A Corma. On the preferential location of al and proton siting in zeolites: A computational and infrared study. Journal of Physical Chemistry, 106, 701-708 (2002).
- Schenter, G, S Kathmann, and B Garrett. Dynamical benchmarks of the nucleation kinetics of water. Journal of Chemical Physics, 116(10), 4275-4280 (2002).
- Schenter, G, S Kathmann, and B Garrett. Equilibrium constant for water dimerization: analysis of the partition function for a weakly bound system. Journal of Physical Chemistry A, 106(8), 1557-1566 (2002).
- Schenter, G. The development of effective classical potentials and the quantum statistical mechanical second virial coefficient of water. Journal of Chemical Physics, 117(14), 6573-6581 (2002).
- Sekino H, Y Kengoku, S Sugiki, and N Kurita.  2003.  “Molecular Orbital Analysis Based on Fragment Molecular Orbital Scheme.”  Chemical Physics Letters 378:589-597.
- Shapley, WA and DP Chong. PW86-PW91 Density functional calculation of vertical ionization potentials: Some implications for present-day functionals. International Journal of Quantum Chemistry, 81(1), 34-52 (2001).
- Shen L, J Shen, X Luo, F Cheng, Y Xu, K Chen, E Arnold, J Ding, and H Jiang.  2003.  “Steered Molecular Dynamics Simulation on the Binding of NNRTI to HIV-1 RT.”  Biophysical Journal 84(6):3547-3563.
- Shen, T, K Tai, RH Henchman, and JA McCammon. Molecular dynamics of acetylcholinesterase. Accounts of Chemical Research, 35, 332-340 (2002).
- Shen, TY, K Tai, and JA McCammon. Statistical analysis of the fractal gating motions of the enzyme acetylcholinesterase. Physical Review e, 63, 041902 (2001).
- Shroll, RM and TP Straatsma. Molecular structure of the outer bacterial membrane of Pseudomonas aeruginosa via classical simulation. Biopolymers, 65, 395-407 (2002).
- Silva, T, J Miller, and TP Straatsma. Revisiting the structural flexibility of the complex p21ras-GTP: the catalytic conformation of the molecular switch II. Proteins: Structure, Function, and Genetics, 45(4), 297-312 (2001).
- Skurski, P, J Rak, J Simons, and M Gutowski. Quasidegeneracy of zwitterionic and canonical tautomers of arginine solvated by an excess electron. Journal of the American Chemical Society, 123(44), 11073-11074 (2001).
- Skylaris C, O Igglessi-Markopoulou, O Detsi, and A Markopoulos.  2003.  “Density functional and Ab Initio Study of the Tautomeric Forms of 3-acetyl Tetronic and 3-Acetyl Tetramic Acids.”  Chemistry of Materials :355-363.
- Smith D, M Dupuis, E Vorpagel, and T Straatsma.  2003.  “Characterization of Electronic Structure and Properties of a Bis(Histidine) Heme Model Complex.”  Journal of the American Chemical Society 125(9):2711-2717.
- Soares, TA, RD Lins, TP Straatsma, and JM Briggs. Internal dynamics and ionization states of the macrophage migration inhibitory factor: Comparison between wild-type and mutant forms. Biopolymers, 65, 313-323 (2002).
- Song, JY, LR Corrales, G Kresse, and H Jonsson. Migration of O vacancies in alpha-quartz: The effect of excitons and electron holes. Physical Review B, 2001. 6413(13): p. art-134102
- Szczepanski, J, J Banisaukas, M Vala, and S Hirata. Preresonance Raman spectrum of the C13H9 fluorene-like radical. Journal of Physical Chemistry A, 106(30), 6935-6940 (2002).
- Szczepanski, J, J Banisaukas, M Vala, S Hirata, R Bartlett, and M Head-Gordon. Vibrational and electronic spectroscopy of acenaphthylene and its cation. Journal of Physical Chemistry A, 106(1), 63-73 (2002).
- Tai, K, T Shen, RH Henchman, Y Bourne, P Marchot, and JA McCammon. Mechanism of acetylcholinesterase inhibition by fasciculin: A 5-ns molecular dynamics simulation. Journal of the American Chemical Society, 124, 6153-6161 (2002).
- Tai, K, TY Shen, U Borjesson, M Philippopoulos, and JA McCammon. Analysis of a 10-ns molecular dynamics simulation of mouse acetylcholinesterase. Biophysical Journal, 81(2), 715-724 (2001).
- Thompson, J, JD Xidos, TM Sonbuchner, et al. More reliable partial atomic charges when using diffuse basis sets. Physical Chemistry Community, 117-134 (2002).
- Tobita M, S Hirata, and R Bartlett.  2003.  “The Analytical Energy Gradient Scheme in the Gaussian Based Hartree-Fock and Density Functional Theory for Two-Dimensional Systems Using Fast Multipole Method.”  Journal of Chemical Physics 118(13):5776-5792.
- Ugliengo, P and A Damin. Are dispersive forces relevant for CO adsorption on the MgO(001) surface? Chemical Physics Letters, 366, 683-690 (2002).
- Vahtras, O, M Engstrom, and B Schimmelpfennig. Electronic g-tensors obtained with the mean-field spin orbit Hamiltonian. Chemical Physics Letters, 351, 424-430 (2002).
- Vainrub, A and BM Pettitt. Surface electrostatic effects in oligonucleotide microarrays: control and optimization of binding thermodynamics. Biopolymers, (2002).
- Valiev M, E Bylaska, and J Weare.  2003.  “Calculations of the Electronic Structure of 3d Transition Metal Dimers with Projector Augmented Plane Wave Method.”  Journal of Chemical Physics 119(12):5955-5964.
- Valiev M, R Kawai, JA Adams, and JH Weare.  2003.  “The Role of the Putative Catalytic Base in the Phosphoryl Transfer Reaction in a Protein Kinase:  First Principles Calculations.”  Journal of the American Chemical Society 125(33):9926-9927.
- Valiev, M, EJ Bylaska, A Gramada, and JH Weare. “First principles molecular dynamics.” In Reviews in modern quantum chemistry: A celebration of the contributions of RG Parr, ed. KD Sen, pp.  1684-1734. World Scientific, Singapore (2002).
- Valiev, M, EJ Bylaska, and JH Weare Bonding structure of 3d transition metal dimers with the projector augmented plane wave method. Journal Of Physical Chemistry, (2002).
- Van Ginhoven RM, H Jonsson, KA Peterson, M Dupuis, and LR Corrales.  2003.  “An Ab Initio Study of Self-Trapped Excitons in Alpha-Quartz.”  Journal of Chemical Physics 118(14):6582-6593.
- van Mourik, T and LEV Emson. A theoretical study of the conformational landscape of serotonin. Phys. Chem. Chem. Phys., 4, 5863-5871 (2002).
- van Mourik, T, GJ Harris, OL Polyansky, J Tennyson, AG Csaszar, and PJ Knowles. Ab initio global potential, dipole, adiabatic, and relativistic correction surfaces for the HCN-HNC system. Journal of Chemical Physics, 115(8), 3706-3718 (2001).
- van Mourik, T. A theoretical study of uracil-(H2O)n, n=2 to 4. Physical Chemistry Chemical Physics, 3(14), 2886-2892 (2001).
- Vargas, R, J Garza, B Hay, and D Dixon. Conformational study of the alanine dipeptide at the MP2 and DFT levels. Journal of Physical Chemistry A, 106(13), 3213-3218 (2002).
- Vargas, R, J Garza, R Friesner, H Stern, B Hay, and D Dixon. Strength of the N-H···O=C and C-H···O=C bonds in formamide and N-methylacetamide dimers. Journal of Physical Chemistry A, 105(20), 4963-4968 (2001).
- Wennmohs F, V Staemmier, and M Schindler.  2003.  “Theoretical Investigation of Weak Hydrogen Bonds to Sulfur.”  Journal of Chemical Physics 119(6):3208-3218.
- White MD, and BP McGrail.  2003.  “Numerical Investigations of Multifluid Hydrodynamics During Injection of Supercritical CO2 into Porous Media.”  In 6th International Conference on Greenhouse Gas Control Technologies (GHGT-6), vol. I and II, ed. J Gale and Y  Pergamon, Kidlington, Oxford, Kaya, Kyoto, Japan.
- White, M and A Ward.. Numerical investigations of vadose zone transport of saturated sodium thiosulfate solutions. American Geological Union 2001 Fall Meeting, San Francisco, CA (2001).
- Windus TL, EJ Bylaska, M Dupuis, S Hirata, LA Pollack, DM Smith, T Straatsma, and E Apra.  2003.  “NWChem:  New Functionality.”  In Proceedings of Computational Science, International Conference on Computational Science 2003, vol. 2660, Springer-Verlag, Berlin, Germany.
- Wirstam, M, SJ Lippard, and RA Friesner. Reversible dioxygen binding to hemerythrin. Journal of the American Chemical Society, (2002).
- Wong, K-Y and BM Pettitt. A study of DNA tethered to a surface by an all-atom molecular dynamics simulation. Theoretical Chemistry Accounts, 106, 233-235 (2001).
- Wu Q, P Ayers, and W Yang.  2003.  “Density-Functional Theory Calculations with Correct Long-Range Potentials.”  Journal of Chemical Physics 119(6):2978-2990.
- Xantheas, S, C Burnham, and R Harrison. Development of transferable interaction models for water: II. Accurate energetics of the first few water clusters from first principles. Journal of Chemical Physics, 116(4), 1493-1499 (2002).
- Yabusaki, S. Multiphase fluid flow and multicomponent reactive transport at the Hanford SX Tank Farm. IAHR International Groundwater Symposium - Bridging the Gap between Measurement and Modeling in Heterogeneous Media. Berkeley, CA (2002).
- Yamataka, H, M Aida, and M Dupuis. Analysis of borderline substitution/electron transfer pathways from direct ab initio MD simulations. Chemical Physics Letters, 353(3-4), 310-316 (2002).
- Yang X, XB Want, and L Wang.  2003.  “On the Electronic Structures of Gaseous Transition Metal Halide Complexes, FeX4- and MX3- (M=Mn, Fe, Co, Ni, X=C1, Br) Using Photoelectron Spectroscopy and Density Functional Calculations.”  Journal of Chemical Physics 119(16):8311-8320.
- Yang, X, X-B Wang, S Niu, CJ Pickett, TI Ichiye, and L-S Wang. Coulomb- and antiferromagnetic-induced fission in doubly charged cubelike Fe-S clusters. Phys. Rev. Letters, 89, 163401 (2002).
- Zaragoza, IP and R Santamaria. The cracking of n-heptane in the gas phase state and in the HZSM-5 zeolite: A quantum molecular dynamics study. Molecular Physics, 100, 3139-3145  (2002).
- Zhai H, B Kiran, and L Wang.  2003.  “Electronic and Structural Evolution of Monoiron Sulfur Clusters, FeSn- and FeSn (n=1-6), from Anion Photoelectron Spectroscopy.”  Journal of Physical Chemistry A 107(16):2821-2828.
- Zhai H, K Boggavarapu, J Li, and L Wang.  2003.  “Hydrocarbon Analogs of Boron Clusters:  Planarity, Aromaticity, and Antiaromaticity.”  Nature Materials 2(12):827-833.
- Zhai H, L Wang, AB Alexandrov, and VG Zakrzewski.  2003.  “A Photoelectron Spectroscopy and Ab Initio Study of B3- and B4- Anions and Their Neutrals.”  Journal of Physical Chemistry A 107(44):9319-9328.
- Zhan, C and D Dixon. Absolute hydration free energy of proton from first principles. Journal of Physical Chemistry A, 105(51), 11534-11540 (2001).
- Zhan, C-G and D Dixon. Electronic excitations in pyrrole: A test case for determination of chromophores in the chromogenic effects of neurotoxic hydrocarbons by time-dependent density functional theory and single-excitation configuration interaction methods. Journal Of Molecular Spectrosc., 216, 81-89 (2002).
- Zhan, C-G and DA Dixon. First-principles determination of the absolute hydration free energy of the hydroxide ion. Journal of Physical Chemistry A, 106, 9737-9744 (2002).
- Zhan, C-G, DA Dixon, MI Sabri, M Kim, and PS Spencer. Theoretical determination of chromophores in the chromogenic effects of aromatic neurotoxicants. Journal of the American Chemical Society, 124(11), 2744-2752 (2002).
- Zhan, C-G, F Zheng, and D Dixon. Electron affinities of Aln clusters and multi-fold aromaticity of the square Al42- structure. Journal of the American Chemical Society, 124(49), 14795-14803 (2002).
- Zhan, C-G, J Gal, and D Dixon. A density functional theory approach to the development of Q-e parameters for the prediction of reactivity in free-radical copolymerizations. Journal of Physical Chemistry A, 106(43), 10311-10325 (2002).
- Zhong S, and JD Fast.  2003.  “An Evaluation of the MM5, RAMS, and Meso-Eta Models at Subkilometer Resolution Using VTMX Field Campaign Data in the Salt Lake Valley.”  Monthly Weather Review 131(7):1301-1322.
- Zimmermann, D, T Haber, H Schaal, and MA Suhm. Hydrogen bonded rings, chains and lassos: The case of t-butyl alcohol clusters. Molecular Physics, 99(5), 413-425 (2001).
- Zverev, V and I Nuretdinov. A quantum-chemical study of the structure of methanofullerene C61H2. Russian Journal Of Physical Chemistry, 76, 1102-1108 (2002).

- Alary F, J Heully, R Poteau, L Maron, G Trinquier, and J Daudey.  2003.  “Using Effective Group Potential Methodology for Predicting Organometallic Complex Properties.”  Journal of the American Chemical Society 125(36):11051-11061.
- Aprà E, E Bylaska, D Dean, A Fortunelli, F Gao, P Krstic, J Wells, and T Windus.  2003.  “NWchem for Materials Science.”  Computational Materials Science 28(2):209-221.
- Bacon DH, and BP McGrail.  2003.  “Waste Form Release Calculations for Performance Assessment of the Hanford Immobilized Low-Activity Waste Disposal Facility Using a Parallel, Coupled Unsaturated Flow and Reactive Transport Simulator.”  In Materials Research Society Symposium Proceedings, Chapter II 1.9, vol. 757, ed. Robert J. Finch and Daniel B. Bullen.  Materials Research Society, Warrendale, Pennsylvania.
- Bacon, D, B McGrail, V Freedman, G Ventura, P Risoluti, and K Krupka. 2002.  Performance assessment of low-level waste disposal facilities using coupled unsaturated flow and reactive transport simulators, in scientific basis for nuclear waste management XXV.  Materials Research Society, Warrendale, PA.
- Baik, M-H, BF Gherman, RA Friesner, and SJ Lippard. Hydroxylation of methane by non-heme diiron enzymes. A molecular orbital analysis of the C-H bond activation by reactive intermediate Q. Journal of the American Chemical Society, 124, 14608-14615 (2002).
- Baik, M-H, D Lee, RA Friesner, and SJ Lippard. Theoretical studies of diiron(II) complexes that model features of the dioxygen-activating centers in non-heme diiron enzymes. Israel Journal of Chemistry, 41, 173-186 (2001).
- Batista, E and RA Friesner. A self-consistent charge-embedding methodology for ab initio quantum chemical cluster modeling of ionic solids and surfaces: application to the (001) surface of hematite (α-Fe2O3). Journal of Physical Chemistry B, 106(33) 8136-8141 (2002).
- Belosludtsev, Y, B Iverson, S Lemeshko, R Eggers, R Wiese, S Lee, T Powdrill, and M Hogan. DNA microarrays based on noncovalent oligonucleotide attachment and hybridization in two dimensions. Analytical Biochemistry, 292(2), 250-256 (2001).
- Bernholdt, DE. 2002. Parallel Computational Chemistry: an Overview of NWChem, in The Sourcebook of Parallel Computing. Morgan Kaufmann Publishers, San Francisco, CA.
- Burnham, C, and S Xantheas. Development of transferable interaction models for water: I. Prominent features of the water dimer potential energy surface. Journal of Chemical Physics, 116(4), 1479-1492 (2002).
- Burnham, C, and S Xantheas. Development of transferable interaction models for water:  III. Reparameterization of an all-atom polarizable rigid model (TTM2-R) from first principles. Journal of Chemical Physics, 116(4), 1500-1510 (2002).
- Burnham, C, S Xantheas, M Miller, B Applegate, and R Miller. The formation of cyclic water complexes by sequential ring insertion: Experiment and theory. Journal of Chemical Physics, 117(3), 1109-1122 (2002).
- Bursten B, M Drummond, and J Li.  2003.  “The Quantum Chemistry of D- and F-Element Complexes:  From an Approximate Existence to Functional Happiness.”  Faraday Discussions 124:1‐24.
- Bursten, BE and J Li. 2001. The electronic structure of organoactinide complexes via relativistic density functional theory:  Applications to the actinocene complexes an (η8C8H8)2 (An=Th-Am), in computational organometallic chemistry. pp. 345-379. Marcel Dekker, New York, NY.
- Bylaska, E, D Dixon, A Felmy, and P Tratnyek. One-electron reduction of substituted chlorinated methanes as determined from ab initio electronic structure theory. Journal of Physical Chemistry A, 106(47), 11581-11593 (2002).
- Bylaska, EJ, M Valiev, R Kawai, and JH Weare. Parallel implementation of the projector augmented plane wave method for charged systems. Computer Physics Communications, 143, 11-
- Cagin, T, G Wang, R Martin, G Zamanakos, N Vaidehi, DT Mainz, and WAG III. Multiscale modeling and simulation methods with applications to dendritic polymers. Comput. Theor. Polyms, 11(5) 345-35628 (2002).
- Campbell, J, H Mercier, H Franke, D Santry, D Dixon, and G Schrobilgen. Syntheses, crystal  structures, and density functional theory calculations of the closo-[1-M(CO)3(η4-E9)]4- (E = Sn, Pb; M = Mo, W) cluster anions and solution NMR spectroscopic characterization of [1-M(CO)3(η4-Sn9)]4- (M = Cr, Mo, W). Inorganic Chemistry, 41(1), 86-107 (2002).
- Chaban GM, SS Xantheas, and RB Gerber.  2003.  “Anharmonic Vibrational Spectroscopy Complexes, n=1, 2.”  Journal of Physical Chemistry A 107(24):4952-4956.
- Chartier, A, C Meis, W Weber, and L Corrales. Theoretical study of disorder in Ti-substituted La2Zr2O7. Physical Review. B, Condensed Matter, 65(13), 134116, 1-11 (2002).
- Chen Y, D Hu, E Vorpagel, and H Lu.  2003.  “Probing Single-Molecule T4 Lysozyme Conformational Dynamics by Intramolecular Fluorescence Energy Transfer.”  Journal of Physical Chemistry B 107: 7947-7956.
- Chipman, D and M Dupuis. Implementation of solvent reaction fields for electronic structure. Theoretical Chemistry Accounts, 107(2), 290-102 (2002).
- Choi Y, and YS Lee.  2003.  “Spin-Orbit Density Functional Theory Calculations for Heavy Metal Monohydrides.”  Journal of Chemical Physics 119(4):2014-2019.
- Choi Y, C Bae, YS Lee, and S Leet.  2003.  “Spin-Orbit Density Functional Theory Calculations for TlAt with Relativistic Effective Core Potentials.”  Bulletin of the Korean Chemical Society 24(6):728‐730.
- Cioslowski J, N Rao, K Pernal, and D Moncrieff.  2003.  “Endohedral Motions Inside Capped Single-Walled Carbon Nanotubes.”  Journal of Chemical Physics 118(10):4456-4462.
- Cioslowski, J, N Rao, and D Moncrieff. Electronic structures and energetics of [5,5] and [9,0] single-walled carbon nanotubes. Journal of the American Chemical Society, 124, 8485-8489  (2002).
- Dabkowska, I, J Rak, and M Gutowski. Computational study of hydrogen-bonded  complexes between the most stable tautomers of glycine and uracil. Journal of Physical Chemistry A, 106(32), 7423-7433 (2002).
- Dang, L and T Chang. Molecular mechanism of ion binding to the liquid/vapor interface of water. Journal of Physical Chemistry B, 106(2), 235-238 (2001).
- Dang, L, T Chang, and A Panagiotopoulos. Gibbs ensemble Monte Carlo simulations of coexistence properties of a polarizable potential model of water. Journal of Chemical Physics,  117(7), 3522-3523 (2002).
- Dang, L. Computational studies of ions binding to the liquid interfaces of water. Journal of Physical Chemistry B, 106(40), 10388-10394 (2002).
- de Jong, W, L Visscher, and W Nieuwpoort. Corrigendum to 'On the bonding and the electric field gradient of the uranyl ion' [Journal Molecular Structure (Theochem,) 548 41-52 (1999)]. Journal Molecular Structure (Theochem), 581, 259 (2002).
- de Jong, W, R Harrison, J Nichols, and D Dixon. Erratum: fully relativistic correlated benchmark results for uranyl and a critical look at relativistic effective core potentials for uranium. Theoretical Chemistry Accounts, 107, 318 (2002).
- de Jong, W, R Harrison, J Nichols, and D Dixon. Fully relativistic correlated benchmark results for uranyl and a critical look at relativistic effective core potentials for uranium. Theoretical Chemistry Accounts, 107(5), 318 (2002).
- Disselkamp, R and M Dupuis. A temperature-dependent study of the ozonolysis of propene.  Journal of Atmospheric Chemistry, 40(3), 231-245 (2001).
- Dixon, D and K Peterson. Heats of formation of CCl and CCl2 from ab initio quantum chemistry. Journal of Chemical Physics, 115(14), 6327-6329 (2001).
- Dixon, D, W de Jong, K Peterson, and J Francisco. Heats of formation of CBr, CHBr and CBr2 from ab initio quantum chemistry. Journal of Physical Chemistry A, 106, 4725-4728 (2002).
- Dixon, DA, D Feller, C Zhan, and J Francisco. Decomposition pathways of peroxynitrous acid: gas phase and solution energetics. Journal of Physical Chemistry A, 106(13), 3191-3196 (2001).
- Doerksen R, B Chen, J Yuan, JD Winkler, and ML Klein.  2003.  “Novel Conformationally-Constrained -Peptides Characterized by 1H NMR Chemical Shifts.”  Chemical Communications :2534‐2535.
- Ducere, J-M, C Lepetit, PG Lacroix, J-L Heully, and R Chauvin. Quadratic hyperpolarizability of carbomeric structures. Chemistry of Materials 14(8), 3332-3338 (2002).
- Dunning, T, R Harrison, D Feller, and S Xantheas. Promise and challenge of high-performance computing, with examples from molecular modeling. Philosophical Transactions. Mathematical, Physical & Engineering Sciences, 360(1795), 1079-1105 (2002).
- Dupuis M, GK Schenter, BC Garrett, and EE Arcia.  2003.  “Potentials of Mean Force with Ab Initio Mixed Hamiltonian Models of Solvation.”  THEOCHEM:  Journal of Molecular Structure 632(1‐3):173‐183.
- Dupuis, M, M Aida, Y Kawashima, and K Hirao. A polarizable mixed Hamiltonian model of electronic structure for micro-solvated excited states: I. Energy and gradients formulation and application to formaldehyde. Journal of Chemical Physics, 117(3), 1242-1255 (2002).
- Dupuis, M, Y Kawashima, and K Hirao. A polarizable mixed Hamiltonian model of electronic structure for solvated excited states: II. Application to the blue shift of the H2CO 1(π* ← n) excitation in water. Journal of Chemical Physics, 117(3), 1256-1268 (2002).
- Dyall, KG. A systematic sequence of relativistic approximations. Journal of Computational Chemistry, 23, 786-793 (2002).
- Dyall, KG. Relativistic and nonrelativistic energy-optimized polarized triple zeta basis sets for the 4p, 5p and 6p elements. Theoretical Chemistry Accounts, 108(6), 335-340 (2002).
- Dyer, K, J Perkyns, and BM Pettitt. Computationally useful bridge diagram series. III. Lennard-Jones mixtures. Journal of Chemical Physics, 116, 9413-9421 (2002).
- El-Azab, A, S Gan, and Y Liang. Binding and diffusion of Pt nanoclusters on anatase TiO2(001)-(1x4) surface. Surface Science, 506(1-2) 93-104 (2002).
- Fann, G, D Jones, E Jurrus, B Moon, and K Perrine. 2002. A parallel computational environment for imaging science. SPIE's 47th Annual Meeting, The International Symposium on Optical Science and Technology. Proceedings of Application of Digital Image Processing, XXV, Seattle, WA.
- Feller D, K Peterson, W deJong, and D Dixon.  2003.  “Performance of Coupled Cluster Theory in Thermochemical Calculations of Small Halogenated Compounds.”  Journal of Chemical Physics 118(8):3510-3522.
- Feller, D, and DA Dixon. Metal ion binding: an electronic structure study of M+(dimethyl ether)n, M = Cu, Ag, and Au and (n = 1-4), complexes. Journal of Physical Chemistry A, 106, 5136-5143 (2002).
- Felmy, AR, H Cho, JR Rustad, and MJ Mason. An aqueous thermodynamic model for polymerized silica species to high ionic strength. Journal Of Sol. Chemistry, 30, (2001).
- Gao, F, E Bylaska, W Weber, and L Corrales. Ab initio and empirical potential studies of defect properties in 3C-SiC. Physical Review. B, Condensed Matter, 64(24), 245208, 1-7 (2001).
- Gervasio, FL, R Chelli, and P Procacci. Is the T-shaped toluene dimer a stable intermolecular complex? Journal of Physical Chemistry A, 106, 2945-2948  (2002).
- Gervasio, FL, R Chelli, P Procacci, and V Schettino. The nature of intermolecular interactions between aromatic amino acid residues. Proteins, 48, 117-125 (2002).
- Ghan, S, N Laulainen, R Easter, R Wagener, S Nemesure, E Chapman, Y Zhang, and R Leung. Evaluation of aerosol direct radiative forcing in MIRAGE. Journal of Geophysical Research, 106, 5295-5334 (2001).
- Ghan, SJ, RC Easter, E Chapman, H Abdul-Razzak, Y Zhang, R Leung, N Laulainen, R Saylor, and R Zaveri. A physically-based estimate of radiative forcing by anthropogenic sulfate aerosol. Journal of Geophysical Research, 106, 5279-5294 (2001).
- Ghan, SJ, X Bian, AG Hunt, and A Coleman. The thermodynamic influence of subgrid orography in a global climate model. Climate Dynamics, 20(1), 31-44 (2002).
- Gherman, BF, BD Dunietz, DA Whittington, SJ Lippard, and RA Friesner. Activation of the C-H bond of methane by intermediate Q of methane monoozygenase: A theoretical study. Journal of the American Chemical Society, 123(16), 3836-3837 (2001).
- Goddard, WA, T Cagin, M Blanco, et al. Strategies for multiscale modeling and simulation of organic materials:  polymers and biopolymers. Comput. Theor. Polyms, 11(5), 329-343 (2001).
- Gramada, A, JH Weare, and EJ Bylaska. Molecular dynamics simulation of the electron solvation in big water clusters. Maui High Performance Computing Center Application Briefs, (2001)
- Guallar, V, BF Gherman, SJ Lippard, and RA Friesner. Quantum chemical studies of methane monooxygenase: Comparison with P450. Computational Bioinorganic Chemistry II, 236-242 (2001). Current Opinion in Chemical Biology, 6(2), 236-242  (2002).
- Guallar, V, BF Gherman, WH Miller, SJ Lippard, and RA Friesner. Dynamics of alkane hydroxylation at the non-heme diiron center in methane monooxygenase. Journal of the American Chemical Society, 124, 3377-3384 (2001).
- Gutowski, M, C Hall, L Adamowicz, J Hendricks, HD Clercq, S Lyapustina, J Nilles, S Xu, and K Bowen. Solvated electrons in very small clusters of polar molecules: (HF)3-. Physical Review Letters, 88(14), art. no. 143001 (2002).
- utowski, M, I Dabkowska, J Rak, S Xu, J Nilles, D Radisic, and K Bowen. Barrier-free intermolecular proton transfer in the uracil-glycine complex induced by excess electron attachment. European Physical Journal D, 20(3), 431-439 (2002).
- Hay, B, D Dixon, J Bryan, and B Moyer. Crystallographic evidence for oxygen acceptor directionality in oxyanion hydrogen bonds. Journal of American Chemical Society, 124(2), 182-183 (2001).
- Heully, J, R Poteau, S Berasaluce, and F Alary. The effective group potential, a new method for the study of spectrum in large molecules: tests and perspectives. Journal Of Chemical Physics, 116, 4829-4836 (2002).
- Hirata S, C Zhan, E Aprà, T Windus, and D Dixon.  2003.  “A New, Self-Contained Asymptotic Correction Scheme to Exchange-Correlation Potentials for Time-Dependent Density Functional Theory.”  Journal of Physical Chemistry A 107:10154-10158.
- Hirata S, M Head-Gordon, J Szczepanski, and M Vala.  2003.  “Time-Dependent Density Functional Study of the Electronic Excited States of Polycyclic Aromatic Hydrocarbon Radical Ions.”  Journal of Physical Chemistry 107(24):4940-4951.
- Ilton E, W deJong, and P Bagus.  2003.  “Intra Atomic Many-Body Effects in P-Shell Photoelectron Spectra of Cr3+ Ions.”  Physical Review. B:  Condensed Matter and Materials Physics 68(12):125106.
- Jones D, E Jurrus, B Moon, and K Perrine.  2003.  “Gigapixel-Size Real-Time Interactive Image Processing with Parallel Computers.”  In 8th International Workshop on High-Level Parallel Programming Models and Supportive Environments, Nice, France, IEEE Computer Society Press, Los Alamitos, California.
- Li J, X Li, H Zhai, and L Wang.  2003.  “Au20:  A Tetrahedral Cluster.”  Science 299(7):864-867.
- Juárez, LH, R Glowinski, and BM Pettitt. Numerical simulation of the sedimentation of a tripole-like body in an incompressible viscous fluid. Applied Mathematics Letters, 15, 743-747 (2002).
- Kaminski, GA, HA Stern, BJ Berne, RA Friesner, YX Cao, RB Murphy, R Zhou, and TA Halgren. Development of a polarizable force field for proteins via ab initio quantum chemistry: First generation model and gas phase tests. Journal Of Computational Chemistry, 23(16), 1515-1531 (2002).
- Kaneti J, SM Bakalova, and M Nguyen.  2003.  “The Ring Closure of Ethylene Phosphites is a New P(III)-Insertion Reaction.  A Computational Study.”  THEOCHEM:  Journal of Molecular Structure 633(1):35-48.
- Kathmann, S and B Hale. Monte Carlo simulations of small sulfuric acid - water clusters. Journal of Physical Chemistry B, 105(47), 11719-11728 (2001).
- Kathmann, S, G Schenter, and B Garret. Understanding the sensitivity of nucleation kinetics: A case study on water. Journal of Chemical Physics, 116(12), 5046-5057 (2002).
- Kawashima, Y, M Dupuis, and K Hirao. Monte Carlo micro-solvation simulations for excited states using a mixed-Hamiltonian model with polarizable and vibrating waters: Application to the blueshift of the H2CO 1(π*←n) excitation. Journal of Chemical Physics, 117, 248-257 (2002).
- Koput, J and K Peterson. The ab initio potential energy surface and vibrational-rotational energy levels of dilithium monoxide, Li20. Journal of Chemical Physics, 116(21), 9255-9260  (2002).
- Koput, J, S Carter, K Peterson, and G Theodorakopoulos. The ab initio potential energy surface and vibration-rotation energy levels of X2Σ+ MgOH. Journal of Chemical Physics, 117(4), 1529-1535 (2002).
- Krstic P, DK Dean, X Zhang, DJ Keffer, Y Leng, PT Cummings, and JC Wells.  2003.  “Computational Chemistry for Molecular Electronics.”  Computational Materials Science :321-341.
- Kruse, P, ER Johnson, GA DiLabio, and RA Wolkow. Patterning of vinylferrocene on H-Si(100) via self-directed growth of molecular lines and STM-induced decomposition. Nano Lett., 2, 807-810 (2002).
- Kurita N, H Inoue, and H Sekino.  2003.  “Adjustment of Perdew-Wang Exchange Functional for Describing Van der Waals and DNA Base-Stacking Interactions 1- 4.”  Chemical Physics Letters :161‐169.
- Li, H, K Yu, EJ Watson, KL Virkaitis, JS D'Acchioli, GB Carpenter, DA Sweigart, PT Czech, KR Overly, and F Coughlin. Models for deep hydrodesulfurization of alkylated benzothiophenes. Reductive cleavage of C-S bonds mediated by precoordination of manganese tricarbonyl to the carbocyclic ring. Organometallics, 21, 1262-1270 (2002).
- Li, J, B Bursten, B Liang, and L Andrews. Noble gas-actinide compounds: Complexation of the CUO molecule by Ar, Kr, and Xe atoms in noble gas matrices. Science, 295(5563), 2242-2245 (2002).
- Li, J. Electronic structures, (d-p)p conjugation effects, and spectroscopic properties of polyoxometalates: M6O192- (M = Cr, Mo, W). Journal of Cluster Science, 13(1), 137-163 (2002).
- Liang B, L Andrews, J Li, and B Bursten.  2003.  “Bonding of Multiple Noble-Gas Atoms to Cuo in Solid Neon:  Cuo(Ng)n (Ng = Ar, Kr, Xe; N = 1, 2, 3, 4) Complexes and the Singlet-Triplet Crossover Point.”  Chemistry:  A European Journal 9(19):4781-4788.
- Liang, B, L Andrews, J Li, and B Bursten. Experimental and theoretical studies of the products of laser-ablated thorium atom reactions with H2O in excess argon. Journal of American Chemical Society, 124, 6723 (2002).
- Liang, B, L Andrews, J Li, and B Bursten. Noble gas-actinide compounds: evidence for the formation of distinct CUO(Ar)4-n(Xe)n and CUO(Ar)4-n(Kr)n (n = 1, 2, 3, 4) complexes.  Journal of the American Chemical Society, 124(31), 9016 (2002).
- Lins, R and T Straatsma. Computer simulations of the lipopolysaccharide membrane of Pseudomonas aeruginosa. Biophysical Journal, 81, 1037-1046 (2001).
- Lumetta, G, B Rapko, P Garza, B Hay, R Gilbertson, T Weakley, and J Hutchison.  Deliberate design of ligand architecture yields dramatic enhancement of metal ion affinity. Journal of the American Chemical Society, 124(20), 5644-5645 (2002).
- Lynch, B and T DG. How well can hybrid density functional methods predict transition state geometries and barrier heights? Journal Of Physical Chemistry A, 105(13), 2936-2941 (2001).
- Malard, JM. Parallel restricted maximum likelihood estimation for linear models with a dense exogenous matrix. Parallel Computing, 28(2), 343-353 (2002).

**Li J, X Li, H Zhai, and L Wang.  2003.  “Au****20****:  A Tetrahedral Cluster.”  ****Science****, 299(7):864-867.**

**Li J, B Bursten, B Liang, and L Andrews. 2002. Noble gas-actinide compounds: Complexation of the CUO molecule by Ar, Kr, and Xe atoms in noble gas matrices. ****Science****, 295(5563), 2242-2245.**

**Ouyang M, JL Huang, CL Cheung, CM Lieber. 2001. Energy gaps in "metallic" single-walled carbon nanotubes. ****Science****, 292(5517), 702-705.**

**Zhai H, K Boggavarapu, J Li, and L Wang.  2003.  “Hydrocarbon Analogs of Boron Clusters:  Planarity, Aromaticity, and Antiaromaticity.”  ****Nature Materials****, 2(12):827-833.**

**Zhan, C-G and DA Dixon. 2002. First-principles determination of the absolute hydration free energy of the hydroxide ion. ****Journal of Physical Chemistry A****, 106, 9737-9744.**

**Breadth of MSCF Scientific Impact***233 publications in the past two years*

**Notes:**
Journal covers, left to right:
Dang LX and TM Chang. 2002. “Molecular Mechanism of Ion Binding in the Liquid/Vapor Interface of Water.” Journal of Physical Chemistry B 106(2):235-238.

Ruscic B, AF Wagner, LB Harding, RL Asher, D Feller, DA Dixon, KA Peterson, Y Song, X Qian, CY Ng, J Liu, W Chen, and DW Schwenke. 2002. Journal of Physical Chemistry A 106(11):2727-2747.

Zhan CG and DA Dixon. 2003. “The Nature and Absolute Hydration Free Energy of the Solvated Electron in Water.” Journal of Physical Chemistry B 107(18):4403-4417.

EK Wilson. 2000. “Chemists Turned Visionaries.” Chemical & Engineering News 78(17):39-45. 

## Proper acknowledgement of MSCF

- Publications or presentations that result from research which has used resources of the MSCF should include the following acknowledgement:
*This research was performed in part using the Molecular Science Computing Facility (MSCF) in the William R. Wiley Environmental Molecular Sciences Laboratory, a national scientific user facility sponsored by the U.S. Department of Energy's Office of Biological and Environmental Research and located at the Pacific Northwest National Laboratory. Pacific Northwest is operated for the Department of Energy by Battelle.*
- The short version of the MSCF Acknowledgement is to be used only for short communications or letters where space is limited to 4 pages or less.
*This research was performed in part using the MSCF in EMSL, a national scientific user facility sponsored by the U.S. DOE, OBER and located at PNNL **Details on MSCF web site.*
- Please send MSCF publication references or hardcopies
-  For details see http://mscf.emsl.pnl.gov/about/acknowledgement.shtml

## MSCF 2005 Call-for-Proposals

**Important Dates**
- Official announcement of call, March 1, 2005
- Letter-of-Intent Due, April 16, 2004
- Full proposals due, May 31, 2004
- Announce awards, September 1, 2004
- Allocations start, October 1, 2004
**Details on MSCF web site**
- http://mscf.emsl.pnl.gov

## MSCF Web site http://mscf.emsl.pnl.gov

**MSCF Web site **_**[http://](http://mscf.emsl.pnl.gov/)**__**[mscf.emsl.pnl.gov](http://mscf.emsl.pnl.gov/)**_

## MSCF Scientific Consulting

- Work with users of the MSCF to utilize our computational resources effectively & efficiently
- Provide computational science expertise
- Train new users of the MSCF Software (MS3)
- Answer MSCF 3rd party software questions
- Scientific visualization resources
- Parallel programming expertise
- Multimedia Resources
- MSCF Web Page
- Workstation training facility

## MSCF Scientific Consulting Staff

- Tina Foley (Just started)
- Group Admin

## Consulting - Getting in touch

- Web access forms:
- http://mscf.emsl.pnl.gov
- Pick “Help” at top of page
- Email (preferred):
- mscf-consulting@emsl.pnl.gov
- Phone:
- 509 – 376 -1301

## Session 1:Getting access and loginMPP2 overviewRunning jobsAllocations and accountingUseful commands and tipsSupportAvailable software

**Session 1****:****Getting access and login****MPP2 overview****Running jobs****Allocations and accounting****Useful commands and tips****Support**Available software

## Accounts/How to Gain Access

- All users have to be associated with a “project”
- All users have to go through a security check
- Complete and return Information Sheets
- Foreign National
- U.S. Citizen Information Sheet
- Review Security Handbook
- Sign and return security agreement training acknowledgement
- Remote users go through same process as visitors to EMSL/PNNL
- Few days to months (depending on country of origin)

## Accounts/Logging in the first time

- Have to use SSH (Secure Shell) Version 2
- System name is “mpp2.emsl.pnl.gov”
- After you receive you userid and initial password
- Login from UNIX
- ssh *<your_userid>*@mpp2.emsl.pnl.gov
- ssh -2 *<your_userid>*@mpp2.emsl.pnl.gov
- Login from PC/Windows
- Point to “mpp2.emsl.pnl.gov”
- Dialogue Box, type in userid and password

## MPP2 overview

- Machine specifics
- 980 HP Longs Peak nodes or 1960 processors
- Dual 1.5 GHz Intel Itanium-2 processors (Madison)
- 11.8 TFlops raw performance
- Quadrics QSNET/Elan interconnect
- OS is Linux based on Red Hat Linux Advanced Server
- 940 nodes, 1880 processors for user jobs
- 574 FatNodes (1148 processors)
- 4 Gbyte RAM / processor (8 Gbyte total)
- 430 GByte local /scratch disk
- 366 ThinNodes (732 processors)
- 3 Gbyte RAM / processor (6 Gbyte total)
- 11 GByte local /scratch disk
- 53 Tbyte global file system running Lustre

## File systems on MPP2

- File systems available to users
- NFS file system /home/*<username> *contains users’ home directory
- /scratch local file system on each node
- Gets cleaned after job finishes
- 430 Gbyte per FatNodes
- 11 Gbyte per ThinNodes
- /dtemp (53 Tbyte) global file system
- Accessible by all nodes
- Volatile (gets cleaned on regular basis, no backups)
- Long term storage on NWfs
- Access via sftp, scp
- You will need an account on this machine

## MPP2 vs MPP1 performance wise

- I.e., 64 processors on MPP2 can do an 12 x larger calculation compared to 64 processors on MPP1!!
- Use the larger memory and disk to your advantage

|  | MPP2 |  | MPP1 | Factor |
| --- | --- | --- | --- | --- |
| # of processors
Peak processor
Peak system | 1880 (940 nodes)
6 Gflops
11.8 Tflops |  | 512 
0.5 Gflops
1⁄4 Tflops |     3+
    12
    45+ |
|  | ThinNodes | FatNodes |  |  |
| Memory per proc.
/scratch per proc. | 3 Gbyte
11 Gbyte | 4 Gbyte
215 Gbyte | 1⁄2 Gbyte
4 Gbyte | 6 to 8
3 to 54 |

## Running jobs

- Building job scripts, we go into some detail
- Running in batch
- Running interactively
- Some paths you might want to include in your .login or .cshrc/.bashrc:
- /home/scicons/scripts 	: contains some tools to build job scripts
- /home/scicons/apps    	: contains supported software
- /home/scicons/libs      	: contains supported libraries

## Constructing a run script for a batch job (1)

- 							    Submission section
- 							    User specific section
- One can also use the bash shell

- #!/bin/csh
- #BSUB –P *<your qbank account*>
- #BSUB –W *<wall clock limit*>
- #BSUB –n *<number of processors>*
- #BSUB –m *<type of processor>*
- #BSUB –o *<your output file>*
- #BSUB –e *<your error file>*
- #BSUB –J *<job name>*
- #BSUB –u *<your email>*
- #BSUB -N
- User defined and job specific section

**Constructing a run script for a batch job (1)**

## Submission section

- QBANK account: project bank account
- Wall clock in *<hh:mm>*

- #!/bin/csh
- #BSUB –P *<your qbank account*>
- #BSUB –W *<wall clock limit*>
- #BSUB –n *<number of processors>*
- #BSUB –m *<type of processor>*
- #BSUB –o *<your output file>*
- #BSUB –e *<your error file>*
- #BSUB –J *<jobname>*
- #BSUB –u *<your email>*
- #BSUB –N

## Submission section

- There are 2 processors per node
- Type of processor:
- FatNodes    	(4 Gbyte RAM per processor)
- ThinNodes	(3 Gbyte RAM per processor)

- #!/bin/csh
- #BSUB –P *<your qbank account*>
- #BSUB –W *<wall clock limit*>
- #BSUB –n *<number of processors>*
- #BSUB –m *<type of processor>*
- #BSUB –o *<your output file>*
- #BSUB –e *<your error file>*
- #BSUB –J *<jobname>*
- #BSUB –u *<your email>*
- #BSUB –N

## Submission section (continued)

- Your job ID should be attached to the end of your output and error files by adding **.%J**:
- I.e. *output.%J and error.%J*

**Constructing a run script for a batch job (4)**

- #!/bin/csh
- #BSUB –P *<your qbank account*>
- #BSUB –W *<wall clock limit*>
- #BSUB –n *<number of processors>*
- #BSUB –m *<type of processor>*
- #BSUB –o *<your output file>*
- #BSUB –e *<your error file>*
- #BSUB –J *<jobname>*
- #BSUB –u *<your email>*
- #BSUB -N

## Submission section (continued)

- Job name cannot have spaces
- Put in your email address, –N tells LSF to send you an email when job is finished

**Constructing a run script for a batch job (5)**

- #!/bin/csh
- #BSUB –P *<your qbank account*>
- #BSUB –W *<wall clock limit*>
- #BSUB –n *<number of processors>*
- #BSUB –m *<type of processor>*
- #BSUB –o *<your output file>*
- #BSUB –e *<your error file>*
- #BSUB –J *<jobname>*
- #BSUB –u *<your email>*
- #BSUB -N

## User specific section

**Constructing a run script for a batch job (6)**

- prun –n <# of processors> <your program>
- Run your job on:
- /scratch  	: 430 Gbyte / node (215 Gbyte / proc) on FatNodes
- 			    11 Gbyte / node (   5 Gbyte / proc) on ThinNodes
- /dtemp	:   53 Tbyte
- Don’t forget to copy the data you want to keep back from /scratch or /dtemp, at the end of your job. Data on these systems is volatile !!

## User specific section (continued) 

- User specific section (continued)
- Copying same file to /scratch on all nodes
- or
- Copying files back from remote processors:
- Running unix commands on remote processors:

- prun -n *<#nodes>* -N *<#nodes>* /home/mscf/bin/pcp *<local file> </scratch/remote file>*

- prun  *ls –l /scratch*

- pdsh –f 30 –w `nodes c $LSB_HOSTS` cp *<local file> </scratch/remote file>*

- foreach node ($LSB_HOSTS)
- 	rcp ${node}:*</scratch/remote file> <local file> *
- end

## User specific section (continued)

- Execute a single program or task:
- prun –n *<# of processors>* *<your program name>*
- Execute multiple programs or tasks:
- prun –n *<# of processors>* *<program 1>*
- prun –n *<# of processors>* *<program 2>*
- prun –n *<# of processors>* *<task 3>*
- .....
- Remember: if one prun fails the whole job dies!!

**Constructing a run script for a batch job (8)**

## Submitting your batch job

**Submitting your batch job**

- Congratulations, your batch job is ready to be submitted!
- Submit your job by typing:
** ****	****bsub   <**    *<your batch job name>*
- When you ask for too much time or processors....
- [bert@m977 ~/runs]$ bsub < very_long.bsub
- Too much time (99 hours) requested for job size (258 CPU)
- Time limit of 48 hours for 256 <= nCPU's <= 1024
- Request aborted by esub. Job not submitted.
* *

## Running interactive jobs

**Running interactive jobs**

- For short and small test / debug jobs
- 32 processors are reserved for interactive work
- Reserved nodes are ThinNodes
- Maximum is 8 processors for 30 minutes
- Starting an interactive session:
- bsub –W *<0:mm> *–P *<QBANK account>* -n *<# procs>* –Is  csh
- Some bsub job variables can be set on the command line or as environment variables
- Set LSF_DEFAULTPROJECT = *<QBANK account>*

## llnw for NWChem users

- Type ( /home/scicons/scripts/ ) llnw, helps you build and submit job script:

- NWChem interactive load leveller submission
- -------------------------------------------
- Input file name (~/nwchem.nw): *nwchem.nw*
- Full path for output (~/nwchem.out):
- Total number of processors you want to use (8): *16*
- Which node type would you like to run on (1=any, 2=FatNodes, 3=ThinNodes) (1):* **2*
- Time limit in HH:MM or MM (1:00): *1:00*
- Account to charge: *mscfcons*
- Are you doing a molecular dynamics run? (y/n): *n*
- NWChem executable (/home/nwchem/bin/nwchem):
*<Output showing you the settings you have choosen>*
- OK? (yes): *yes*

## I submitted my job, when will it run?

- The LSF scheduler is used to maximize the utilization of the machine, scheduling jobs and doing backfill
- Scheduler works using 				      first come, first serve strategy
- You can use backfill to your 			     advantage submitting a job small 			      and short enough to run directly
-     (check availability with **window**)

## Job running policies

- No....you cannot run a job on all 1880 processors for a week!!
- Job running policies on MPP2
- Number of processors:
- 	- Minimum of  8, maximum of 1024
- Time limits:
- 	- 8-255 processors 	= 72 wall-clock hours
- 	- 256-1024 processors 	= 48 wall-clock hours
- Large jobs (256+) have higher priority
- Maximum of 3 running jobs and 8 queued jobs
- Interactive or short queue allows you to run test and debug jobs on 1-8 processors for a maximum of 30 minutes

## SIGHTS special purpose queue (1)

- _S_cientific _I_mpact _G_enerated by _H_igh _T_eraflop _S_imulations
- Requirements (both need to be fulfilled):
- Job needs resources beyond normal queue limits
- 1024 – 1880 processors
- Extended run limits
- Will deliver unique high-impact cutting-edge PNNL/EMSL mission science results which cannot be performed at any other computing facility
- Bonus:
- Time used during the run will be reimbursed

## SIGHTS special purpose queue (2)

- Getting a SIGHTS job setup:
- Submit request to _[mscf-consulting@emsl.pnl.gov](mailto:mscf-consulting@emsl.pnl.gov)_ with a short, one page, description of the job and its scientific impact
- Consultants will review the proposal
- Job will be run after a scheduled Thursday outage
- Consultants will help you set up the job and monitor

## Allocations and accounting

- Your allocation of time is in the bank, QBank that is 
- QBank will do the accounting & withdrawals for you
- Commands to query your allocation:
- qbalance -h		: gives balance in cpu-hours
- qlsuser			: gives a list of all accounts 				 		  available to you
- qtxns –u *<username>*	: gives overview of
- 					  all your withdrawals
- Remember: you will be charged for all
- 	processors dedicated to you

## Useful commands (1)

**Useful commands (1)**

- [bert@m977 ~]$ showq
- Jobid User     Stat SubmitTime  cpu RunLim StartTime   Jobname
- ===================================================================
- 35430 kirilt   RUN  02/06-07:24 256 03:53  02/06-11:46
- 35451 jason    RUN  02/06-10:40  32 13:48  02/06-15:41 26736
- 35478 freedman RUN  02/06-12:09 512 01:48  02/06-15:41 glass_test
- 35453 twarth   RUN  02/06-10:43  32 14:41  02/06-16:34 26914
- 35432 morozov  RUN  02/06-07:28  64 06:54  02/06-16:47
- 35487 kwong    RUN  02/06-13:34 128 11:08  02/06-17:01
- 35446 andri    PEND 02/06-10:10  64 10:00
- 35454 luukv    PEND 02/06-10:44 1024 16:00              26993
- 35488 sally    PEND 02/06-13:34 128 06:00
- 35489 cchen    PEND 02/06-13:34 372 04:00
- 35507 nellie   PEND 02/06-15:26  32 16:00              blyp
- 35508 d3k958   PEND 02/06-15:26  64 16:00              b3lyps

## Useful commands (2)

**Useful commands (2)**

## Useful commands (3)

- bjobs alone will give you a list of _**your**_ jobs !

- [bert@m976 ~]$ bjobs -u all
- JOBID   USER     STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
- 35478   freedman  RUN   normal     m977        2*m106      glass_test Feb  6 12:09
-                                                2*m105
- 35487   apostolu  RUN   normal     m976        2*m121      *How many  Feb  6 13:34
-                                                2*m112
-                                                2*m107
-                                                2*m108
-                                                2*m88
-                                                2*m110
-                                                2*m109
-                                                2*m111
- 35446   nchoudhu  PEND  normal     m977                    *129.02.nw Feb  6 10:10
- 35454   dabaker   PEND  normal     m976                    26993      Feb  6 10:44
- 35509   pzhang    PEND  normal     m976                    uhf        Feb  6 15:26
- 35514   rory      PEND  normal     m977                    jobs       Feb  6 17:19

**Useful commands (3)**

## Useful commands (4)

- [bert@m976 ~]$ rinfo
- MACHINE     CONFIGURATION
- m          lsf
- PARTITION       CPUS    STATUS        TIME   TIMELIMIT  NODES
- root            1958                                    m[0-977] mh2
- parallel    1718/1880   running 12:19:41:24              m[0-939]
- lustre          0/68   running 12:19:41:27              m[940-973]
- admin            0/8   running 12:19:41:30              m[974-977]
- RESOURCE        CPUS    STATUS        TIME    USERNAME  NODES
- parallel.18258    64 allocated    06:07:35      twarth  m[16-31,41-47,64,113-120]
- parallel.18300    32 allocated    02:12:53     rinaldo  m[89-104]
- parallel.18301     4 allocated    02:12:53       bckim  m[105-106]
- parallel.18312    64 allocated    01:06:46       kclau  m[40,48-63,65-79]
- parallel.18317    16 allocated       52:37       swang  m[88,107-112,121]
- JOB             CPUS    STATUS        TIME    USERNAME  NODES
- parallel.19100     1   running    06:07:32      twarth  m16
- parallel.19101    64   running    06:07:32      twarth  m[16-31,41-47,64,113-120]
- parallel.19178     1   running    02:11:41     rinaldo  m89

**Useful commands (4)**

## Useful commands (5)

- What happened to my job:** bhist –l *****<jobid>***

- [bert@m977 ~]$ showq
- Jobid User     Stat SubmitTime  cpu RunLim StartTime   Jobname
- =============================================================================
- 35430 twarth   RUN  02/06-07:24  64 03:53  02/06-11:46
- 35451 gokhale  RUN  02/06-10:40  32 13:48  02/06-15:41 26736
- 35478 freedman RUN  02/06-12:09   4 01:48  02/06-15:41 glass_test
- 35453 baxter   RUN  02/06-10:43  32 14:41  02/06-16:34 biosomething
- 35432 dutta    RUN  02/06-07:28  64 06:54  02/06-16:47
- 35487 het      RUN  02/06-13:34  16 11:08  02/06-17:01

**Useful commands (5)**

## Useful commands (6)

- Other info about the machine:
** ****/etc/motd **: contains login message

- [bert@m977 ~]$ window
- Only jobs of 8 CPU’s or less can use Shortpool
- Backfill Window
-   30 processors (  15 nodes) Shortpool Available for 00:30:00
-   14 processors (   7 nodes) FatNodes  Available for 15:49:59
-    8 processors (   4 nodes) FatNodes  Available for 17:00:24
-   24 processors (  12 nodes) ThinNodes Available for 09:04:23
- ----------------------------------------
- For normal jobs    46 processors (  23 nodes) total

**Useful commands (6)**

## When things go wrong?

- MSCF general web site with lots of info
- 	_*http://mscf.emsl.pnl.gov/*_
- Quick reference card
- MSCF Visualization and
- 	User Services group (VisUS)
- For refunds
- General help running jobs
- Assistance getting your code to run (in parallel)
- Send an email to mscf-consulting@emsl.pnl.gov

## Job refunds

- When are you eligible for a refund:
- Hardware failures
- System software failures
- Certain user software cases
- If you want to apply for a refund:
- Send message to MSCF Consulting queue
- Save and provide access to information for verification:
- We need to verify the source of the problem before giving refund
- We will need appropriate input, output, error file, and job ID
- Please make your directory readable for us
- Refund based on time lost; partial refund given in cases where restarts were possible

## Session 1:Getting access and loginMPP2 overviewRunning jobsAllocations and accountingUseful commands and tipsSupportAvailable software

**Session 1****:**Getting access and loginMPP2 overviewRunning jobsAllocations and accountingUseful commands and tipsSupport**Available software**

## Supported program packages

- Program packages on MPP2
- Location is /home/scicons/apps
- Fully supported codes:
- NWChem 4.5 (is located in /home/nwchem/bin)
- MOLPRO (2002.6)
- GAMESS-US (Dec.12, 2003)
- ADF (2003.01)
- Available codes:
- DaCapo
- MM5
- Amber7

## Executables and Scripts

**Location of executables**:
- 			/home/scicons/apps
**Submit scripts**:
- 			/home/scicons/scripts
- 			llnw
- 			submit_molpro
- 			submit_gms
- 			submit_adf
- Hint: add /home/scicons/scripts to your $PATH

## Features of NWChem

- Quantum Methods
- RHF, UHF, ROHF, MP2-4, RI-MP2, CISD, CISDT, CISDTQ, MCSCF, Selected CI
- CCSD, LCCD, LCCSD, CCSD(T), CCSDT, CCSDTQ
- R/UKS LDA, GGA, Hybrid
- CIS, TDHF, TD-DFT and SO-DFT
- PSPW, Band
- Molecular dynamics
- MD, QM/MM, QMD
- Other
- Solvent Effects, Relativistic effects, NMR
- ONIOM
- Ecce interface
- Highly parallelized

## Example for NWChem

-      or
- llnw uo2.nw -accnt gc3 -nprocs 16 -time 0:30 -mdon n -nset 1

- TITLE "uo2(2+) d4h opt freq"
- charge 2
- GEOMETRY units angstrom print
-    U        0.0    0.0    0.00
-    O        0.0    0.0    1.79
-    O        0.0    0.0   -1.79
-    symmetry D4h
- END
- BASIS print
-    U  library stuttgart_rsc_1997_ecp
-    O  library aug-cc-pvtz
- END
- ECP
-    U  library stuttgart_rsc_1997_ecp
- END
- DFT
-    grid xfine
-    print medium
- END
- TASK dft energy
- TASK dft optimize
- TASK dft freq numerical

**Example for NWChem**

**Input file**:
-              uo2.nw
**Submission**:

## Scalability of NWChem on MPP2

**Si****75****O****148****H****66**
- 3554 basis
- functions

## Features of MOLPRO

- Quantum Methods
- R/U/ROHF and DFT
- MP2, MP3, MP4SDQ, MP4, CISD, R/UCCSD, R/UCCSD(T), FCI
- CASSCF, CASPT2, CASPT3, MR-CISD, MR-ACPF, MR-AQCC
- LMP2, LCCSD, LCCSD(T), EOM-CCSD
- Analytic energy gradients for HF, DFT, MP2, LMP2, QCISD, MCSCF, and CASPT2 (soon)
- Numerical vibrational frequencies
- Parallel parts:
- Energy only: SCF, DFT, MRCI, MP2, LMP2, CCSD(T)
- Analytical gradients: SCF, DFT, LMP2

## Example for MOLPRO

-      or
- submit_molpro h2o.inp -a gc3 -n 16 -t 0:30 -q 1

- ***,h2o                   !job title
- geometry={o;              !z-matrix geometry input
-           h1,O,r;
-           h2,O,r,H1,theta}
- r=1.85,theta=104          !set geometry parameters
- basis=vtz                 !use VTZ basis
- hf                        !closed-shell scf
- ccsd(t)                   !do ccsd(t) calculation
- casscf                    !do casscf calculation
- mrci                      !do mrci calculation

**Example for MOLPRO**

**Input file**:
-              h2o.inp
**Submission**:

## Scalability of MOLPRO on MPP2

**RHF-RCCSD(T) **
- 427 Basis Functions

## Features of GAMESS-US

- Quantum Methods
- R/U/ROHF
- DFT: LDA, GGA, Hybrid
- R/U/RO-MP2,CIS, CISD,CCD, CCSD, CCSD(T)
- MCSCF, MC-QDPT, MR-CISD
- Other:
- Relativistic: DK, RESC, NESC, SO-CI, SO-MCQDPT
- VSCF, TDHF, polarizability, hyperpolarizability, NMR, IR, Raman
- SCRF, COSMO, PCM, EFP, QM/MM
- Monte-Carlo global minimization

## Example for GAMESS

-      or
- submit_gms c60.inp -a gc3 -n 16 -t 0:30 -q 1

-  $CONTRL SCFTYP=RHF MPLEVL=2 RUNTYP=GRADIENT $END
-  $SYSTEM TIMLIM=300 MWORDS=300 memddi=400 parall=.true. $END
-  $BASIS  GBASIS=N311 NGAUSS=6 POLAR=POPN311 ndfunc=1 $END
-  $GUESS  GUESS=HUCKEL $END
-  $DATA
- C60 Ih in C1     optimized structure
- C1       1
- C   6.0   0.000000    3.449997    0.684800
- ......
- C   6.0   1.170975   -3.002716   -1.408493
-  $END

**Example for GAMESS**

**Input file**:
-               c60.inp
**Submission**:

## Scalability of GAMESS on MPP2

**RMP2 **
- 302 Basis Functions

## Features of ADF

- Slater Type Orbital based basis sets
- LDA, GGA, Meta-GGA
- TDDFT, polarizability, hyperpolarizability
- QM/MM
- Relativistic options:
- Pauli, ZORA, spin-orbit coupling
- Properties:
- IR, Raman, UV-Vis, MCD, NMR, ESR, COSMO
- Others aspects
- broken-symmetry, multiplets
- Band code

## Example for ADF

-      or
- submit_adf oh.inp -a gc3 -n 32 -t 0:30 -q 1

- adf -n $NPROC  << eor
- TITLE    ADF Calculations for OH radical
- XC
-   LDA vwn
-   GGA pw91
- END
- BASIS
-  type TZ2P
-  core small
- END
- SYMMETRY C(2v)
- ATOMS cartesian
-   O   0.0   0.0   0.0
-   H   0.0   0.0   1.1
- END

**Example for ADF**

**Input file**:  oh.inp
**Submission**:

- GEOMETRY
-   optim cartesian
-   convergence grad=1e-3
-   iterations 30
- END
- DIRECTSCF
- UNRESTRICTED
- CHARGE 0 1
- OCCUPATION
-   sigma 2//2
-   pi    2//1
-   delta 0//0
-   phi   0//0
- END
- end input
- eor

## Scalability of ADF at MPP2

**PW91 **
- 1672 Basis Functions

## Supported libraries

- Supported libraries available on MPP2
- Libraries in /home/scicons/libs
- Global Array tools (GA)
- PETSC 2.1.6
- SCALAPACK 1.7
- NetCDF 3.5.1 (both for Gnu and Intel compiler)
- New libraries and programs can be added on request, depending on general usefulness and support

## BREAK

## Session 2:Compiling and Linking on MPP2Debugging and Profiling on MPP2

**Session 2:****Compiling and Linking on MPP2**Debugging and Profiling on MPP2

## Available compilers

- Intel compiler (USE THIS ONE FOR MPP2!!!)
- Fortran (efc), version 7.1,   	8.0 on the way
- C, C++ (ecc), version 7.1,	8.0 on the way
- /opt/intel/compiler70/ia64/bin
- GNU compiler
- Fortran (f77)
- C, C++ (gcc, g++)
- /usr/bin

## Before you use Intel compiler ...

- Source eccvars.csh or efcvars.csh
-    (or . /eccvars.sh and . /efcvars.sh for bash users)
- Sets PATH, LD_LIBRARY_PATH and MAN_PATH
- Lets you use “ecc” or “efc” without full path
- Located in /opt/intel/compiler70/ia64/bin              	default compiler
- Is set by default on the system

## Some Intel compiler options

- Optimization
- -O0 	(none)
- -O2 (-O)	(_**default**_, assume is SAFE)
- -O3 	(aggressive, potentially UNSAFE)
- -tpp2 	targets the Itanium-2 processor
- -ftz 		flushes denormal results to 0
- Floating point assist problem
- -ipo 	interprocedural optimization across files
- -FR 	free-form FORTRAN (fortran only)
- -r4, -r8, -i4, -i8 set default real and int size (fortran only)
- -Vaxlib 	links in portability library

## Parallel Compiling and Linking on MPP2: {the easy way}

- Use /usr/lib/mpi/mpi_intel/bin/mpicc
- for C and C++
- Use /usr/lib/mpi/mpi_intel/bin/mpif90 or mpif77
- for FORTRAN
- Automatically links to correct MPI libraries and include files

## Parallel Compiling and linking on MPP2:{the whole story}

- Use efc or ecc with parallel includes, libraries
- -I/usr/lib/mpi/mpi_intel/includeXXX	MPI includes
- -L/usr/lib/mpi/mpi_intel/libXXX
- -lmpifarg (mpi_init, etc...)
- -lmpi (mpi_finalize, mpi_recv, mpi_comm_rank, etc...)
- substitute	    to set the default size     compile with
- XXX = “”  	     4 byte reals, 4 byte int
- XXX = “_i8”     4 byte reals, 8 byte int    -i8
- XXX = “_r8”     8 byte reals, 4 byte int    -r8
- XXX = “_i8r8”  8 byte reals, 8 byte int    -r8 –i8

## Other important libraries on MPP2

- -L/usr/lib/
- -lelan (Quadrics QSNET/Elan interconnect specific)
- -L/opt/intel/compiler70/ia64/lib
- -lintrins (intrinsic math functions: floor, etc...)
- -lPEPCF90 (compatibility library)
- -lguide (omp, kmp, wrapper for standard gnu libs: gethostname, getenv, etc...)
- -L/lib
- -lpthread (standard POSIX threads library)

## Math libraries optimized for MPP2

- HP MLIB fast matrix and vector routines
- Contains BLAS 1-3, LAPACK, and other routines
- Link path: -L/opt/mlib/lib/linux
- Use –lveclib8 and –llapack8 with –i8 option
- Use –lveclib and –llapack with –i4 (default) option
- You can use –openmp option for Mlib libraries
- Disable threading by setting env variable
- setenv OMP_NUM_THREADS 1
- setenv MLIB_NUMBER_OF_THREADS 1

-                   For 8 byte integer apps requiring veclib, for instance...
**%> mpif90 –o myexe mycode.f –i8 -openmp –L/opt/mlib/lib/linux –lveclib8**

-  For 4 byte integer apps requiring veclib, for instance...
**%> mpif90 –o myexe mycode.f –openmp –L/opt/mlib/lib/linux –lveclib**

## How to go Wrong at Run Time

- Strange errors if you link with one library set and run with another
- Precedence is left-to-right for –L and –l
- Nested library calls will get tested if dynamic, resolved if static
- -L/usr/lib/mpi/lib -L/usr/lib/mpi/mpi_intel/lib –lmpi
- Will pick up libmpi.so from first directory (the wrong one)
- The correct one is the mpi_intel directory
- -lVT –lmpi     {Vampir trace}
- Will pick up definition for MPI_Init() from libVT.a

*LD_LIBRARY_PATH is different*

## Putting it all together...

- source /opt/intel/compiler70/ia64/bin/efcvars.csh
- setenv LD_LIBRARY_PATH  \ /usr/lib/mpi/mpi_intel/lib_i8r8: \ /usr/lib:/opt/intel/compiler70/ia64/lib:/lib: \ /opt/mlib/lib/linux:$LD_LIBRARY_PATH
- Code is ./mytest.f
**%>** efc -c mytest.f  \
-    **-I/usr/lib/mpi/mpi_intel/include_i8r8** –O2 –ftz \          –tpp2 –i8 –r8 **|& tee mytest_compile.log**
**%>** efc –o mytest mytest.o \
-    **-L/usr/lib/mpi/mpi_intel/lib_i8r8** –lmpifarg –lmpi \
-    **–****L/usr/lib** –lelan **-L/opt/intel/compiler70/ia64/lib** \   –lintrins –lPEPCF90 –lguide **–L/lib** –lpthread       \    **–L/opt/mlib/lib/linux** –lveclib8 **|& tee mytest_link.log**

## If you have compile problems on MPP2

- Which compiler(s) you are using?
- Where your build directory is located
- (make it and the relevant files world-readable so we can see it!)
- Your LD_LIBRARY_PATH
- May be different during a session, so it does not always agree with your .cshrc or .bashrc file.
- The exact error(s) you are receiving during compile/link/run time
- Compile/Build output log files
- With libraries, includes, and compiler version
- “printenv” listing
- ldd *<executable>*

- MSCF consultant support can help given this information:

## Session 2:Compiling and Linking on MPP2Debugging and Profiling on MPP2

**Session 2:**Compiling and Linking on MPP2**Debugging and Profiling on MPP2**

## Debugging Codes on MPP2

**Debugging Codes on MPP2**

- Does code reproduce the error on 8 or less procs?
- Yes: use TotalView.
- http://mscf.emsl.pnl.gov/hardware/totalview.shtml
- Compile and link with the “-g” option and no optimization, i.e. “-O0”
- Open an interactive session on MPP2:
- bsub –n 8 –W 0:30 –P *<QBank acount>* –Is csh
- Execute the commands in your bsub script that come
- 	before your “prun” command.
- Use total view to execute your prun command

- /home/scicons/bin/totalview prun –a –n 8 *<your_program> <your_program’s_input>*

## TotalView continued

- Step 3 times (or until your main program shows up in the “source” window with a yellow stop arrow at your first executable statement)
- When popup asks
**“****Do you want to stop the process now?”**
- Click on the “Yes” button.
- The “Stack Trace” window tracks what your call stack looks like.
- The “Stack Frame” window tracks the active variables and data structures at the current execution point.

## TotalView continued

- Click on line numbers in the source code window to set/unset  break points (they toggle).
- Learn to “dive”. Chapter 1 of the
- 	“TotalView User Guide” (pp. 1-13)
- http://www.etnus.com/Support/docs/index.html
- Some TotalView QuickView manuals available

## NO! My code runs on 8 processors but fails on more

- Take a deep breath. Exhale slowly. Don’t panic.
- Exceptions to the maximum of 8 interactive processors are possible, though strongly discouraged.
- It makes very low utilization of the machine.
- It’s hard to meaningfully sift through that many process windows in TotalView.
- Login on a node your job is running on
- Run TotalView attaching to your process by process id.
- You have to do this while your job is actively running.

## TotalView debugging: Attaching to a running job

**TotalView debugging: Attaching to a running job**

- Use “bjobs” command
- When it says “RUN” under the STAT column, the EXEC_HOST column has a list of nodes you are running on.
- While your job is still running
- Use ssh to open a window on one of your processors From one of the mpp2 login nodes type
- 	ssh  -X m###
- Here ### is a 1 to 3 digit node number.
- -X will forward the display/terminal variables

## TotalView: Attaching to a process id continued

**TotalView: Attaching to a process id continued**

- Once logged determine process id of your running job
- Use “top” or
- “ps –ef | grep*<your_login_id>*”
- Then launch total view with no arguments:
- /home/scicons/bin/totalview
- Click on “File” button.
- Select “New Program” option.
- Click on the “Process ID button” (so it turns green) and type in the process id you wish to attach to.
- “Halt” button stops running code and see where it is.

## Debugging without TotalView

- Think about alternate methods:
- Try gdb or idb (idb can handle parallel debugging)
- Use value from MPI_COMM_RANK to build a unique
-     filename in the tree /dtemp/*<your_login>*
- Open one file for each process.
- Use print (printf) and bisection. Use flush/fflush after each print.
- Find out where the code is breaking.
- Trace the MPI communications around that point.
- Contact the consultants
- (first see _[http://](http://mscf.emsl.pnl.gov/system/tips.shtml)__[mscf.emsl.pnl.gov/system/tips.shtml](http://mscf.emsl.pnl.gov/system/tips.shtml)_
-    “When good programs go bad...”)

## Profiling: Vampir

- Vampir is a visualization and performance analyser tool for MPI programs
- See _[http://mscf.emsl.pnl.gov/hardware/vampir.shtml](http://mscf.emsl.pnl.gov/hardware/vampir.shtml)_
-  source /home/scicons/apps/vampir/vampir.csh
- or
- . /home/scicons/apps/vampir/vampir.sh
- This sets VT_ROOT, PAL_LICENSEFILE, PAL_ROOT and VAMPIR_LIB_PATH environment variables.

## Vampir continued

- Link with vampirtrace and pthread libraries ahead of the mpi library:
- efc –ftz –O2 –tpp2 test_mpi.f90 –o test_mpi  \
- 		-L${VAMPIR_LIB_PATH} –lVT –lpthread  \
- 		-L*<MPI_library_path>* –lmpifarg –lmpi –lelan
- Execute your code in a bsub script/command
-     as you normally would.
- This creates a *<your_executable>*.bvt file in
-     the directory where you run.
- In the example above a test_mpi.bvt file is created

## Vampir continued

- Sometimes you get a .stf file instead of a .bvt file.
-     .stf files can be analyzed.
-   /home/scicons/apps/vampir test_mpi.bvt
-                                  or
-   /home/scicons/apps/vampir test_mpi.stf
- If your code doesn’t call mpi_finalize you won’t get any useful profiling files.
- Remember to copy the files back if you are running from /scratch or /dtemp

## Vampir continued

- Select one node / all of the nodes from the thumbnail sketch (“Frame Display” window)
- Use left mouse clicks on the appropriate node lines.
- Next  select a timing window of interest
- Move mouse pointer to the beginning of time interval you wish to view in the “Frame Display” window
- Hold down the left mouse button and drag to extend the window to the endpoint of the time interval you wish to examine.
- With the mouse pointer in the “Frame Display” window
-  Right click and select “Load” from the menu.

## Vampir continued

- Choose “Selected Time Interval” rather than “All”
- Be very patient, take a break while the profile data is loaded.
- Examine the statistics and zoom in on the timelines for the interval/processors you have loaded profile data for.
- See pages 1-16 in the Vampir User’s Manual.
- _[http://www.pallas.com/e/products/pdf/Vampir-userguide.pdf](http://www.pallas.com/e/products/pdf/Vampir-userguide.pdf)_

## Future tutorials

- Some possible future tutorials
- NWChem and Ecce
- Parallel Programming
- Debugging
- Profiling
- 3rd party software at the MSCF
- Other requests?
- Feedback would be appreciated

## In closing...

- MSCF VisUS Support
- can be reached via email at:
** ****mscf-consulting@emsl.pnl.gov**