tightened tolerance  factors in E-W algorithm
Merge branch 'cekees/fix_moving_mesh' of https://github.com/erdc-cm/proteus into cekees/fix_moving_mesh
added barrier just  to make sure mesh files  have been written before partitioning
Merge branch 'cekees/fix_moving_mesh' of https://github.com/erdc-cm/proteus into cekees/fix_moving_mesh

This commit should give us more detailed info about what is not scaling. For the prior commit, here is the petsc  profile of  the partitioning section for 512 cores and a mesh that looks like:

  Mesh points: 521772
  Mesh tetrahedra: 3159470
  Mesh faces: 6366207
  Mesh edges: 3728507
  Mesh boundary faces: 94534
  Mesh boundary edges: 1466

************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

/var/spool/alps/27091554/parun on a arch-linux2-c-opt named nid05470 with 256 processors, by cekees Tue Sep 23 23:23:34 2014
Using Petsc Release Version 3.4.4, Mar, 13, 2014

                         Max       Max/Min        Avg      Total
Time (sec):           1.592e+03      1.00161   1.590e+03
Objects:              1.200e+01      1.00000   1.200e+01
Flops:                0.000e+00      0.00000   0.000e+00  0.000e+00
Flops/sec:            0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Messages:         3.000e+00      2.00000   2.988e+00  7.650e+02
MPI Message Lengths:  1.657e+06      2.00000   5.523e+05  4.225e+08
MPI Reductions:       2.700e+01      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total
 0:      Main Stage: 1.5322e+02   9.6%  0.0000e+00   0.0%  0.000e+00   0.0%  0.000e+00        0.0%  0.000e+00   0.0%
 1: Mesh Partitioning -- Tetgen Files Algorithm: 1.4368e+03  90.4%  0.0000e+00   0.0%  7.650e+02 100.0%  5.523e+05      100.0%  2.600e+01  96.3%

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

PetscBarrier           1 1.0 5.1107e-0111908.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0

--- Event Stage 1: Mesh Partitioning -- Tetgen Files Algorithm

MatAssemblyBegin       1 1.0 5.0068e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd         1 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatPartitioning        1 1.0 2.0882e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  4   0  0  0  0  4     0
Read elements          1 1.0 1.3668e+01 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
Repartition nodes       1 1.0 9.7829e+00 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+01  1  0  0  0 41   1  0  0  0 42     0
Build subdomains       1 1.0 1.3668e+03 1.0 0.00e+00 0.0 2.6e+02 3.9e+05 5.0e+00 86  0 33 24 19  95  0 33 24 19     0
Build subdomains faces       1 1.0 3.4877e+01 1.0 0.00e+00 0.0 2.6e+02 8.0e+05 5.0e+00  2  0 33 48 19   2  0 33 48 19     0
Build subdomains edges       1 1.0 1.6305e+01 1.0 0.00e+00 0.0 2.6e+02 4.7e+05 5.0e+00  1  0 33 28 19   1  0 33 28 19     0
Build subdomains renumber       1 1.0 5.2405e+00 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
Build subdomains cleanup       1 1.0 4.0439e-01 5.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Viewer     1              0            0     0

--- Event Stage 1: Mesh Partitioning -- Tetgen Files Algorithm

           Index Set     9              9       240968     0
              Matrix     1              0            0     0
 Matrix Partitioning     1              1          644     0
========================================================================================================================
Average time to get PetscTime(): 1.90735e-07
Average time for MPI_Barrier(): 2.49863e-05
Average time for zero size MPI_Send(): 1.77324e-06
-log_summary
-mcorr_ksp_type cg
-mcorr_pc_hypre_type boomeramg
-mcorr_pc_type hypre
-mesh_ksp_type cg
-mesh_pc_asm_type basic
-mesh_pc_type asm
-ncls_ksp_type fgmres
-ncls_pc_hypre_type boomeramg
-ncls_pc_type hypre
-rans2p_ksp_type fgmres
-rans2p_pc_asm_type basic
-rans2p_pc_type asm
-rdls_ksp_type fgmres
-rdls_pc_hypre_type boomeramg
-rdls_pc_type hypre
-vof_ksp_type fgmres
-vof_pc_hypre_type boomeramg
-vof_pc_type hypre
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Fri Aug 29 15:37:00 2014
Configure options: --prefix=/lustre/home1/u/cekees/.hashdist/bld/petsc/l62hty3nzk4z LDFLAGS=-dynamic --with-cmake=/lustre/home1/u/cekees/.hashdist/bld/cmake/zukedra6j3lj/bin/cmake --with-batch --with-clanguage=C --known-level1-dcache-size=16384 --known-mpi-shared-libraries=1 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-c-double-complex=1 --with-pthread=1 COPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=0 --with-blas-dir=/lustre/home1/u/cekees/.hashdist/bld/blas/orqin7luuh5d --with-lapack-dir= --with-metis-dir=/lustre/home1/u/cekees/.hashdist/bld/parmetis/asuw7mtvy2yg --with-parmetis-dir=/lustre/home1/u/cekees/.hashdist/bld/parmetis/asuw7mtvy2yg --with-cmake-dir=/lustre/home1/u/cekees/.hashdist/bld/cmake/zukedra6j3lj --with-mpi-compilers CC=cc CXX=CC F77=ftn F90=ftn FC=ftn --with-patchelf-dir=/lustre/home1/u/cekees/.hashdist/bld/patchelf/m5njcb5667me --with-python-dir=/lustre/home1/u/cekees/.hashdist/bld/python/tlcaadhvtyee --download-ptscotch=1 --download-superlu=1 --download-superlu_dist=1 --download-spooles=1 --download-hypre=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1
-----------------------------------------
Libraries compiled on Fri Aug 29 15:37:00 2014 on garnet01
Machine characteristics: Linux-2.6.32.59-0.7.3-default-x86_64-with-SuSE-11-x86_64
Using PETSc directory: /lustre/home1/u/cekees/.hashdist/tmp/petsc-l62hty3nzk4z
Using PETSc arch: arch-linux2-c-opt
-----------------------------------------

Using C compiler: cc  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O2  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: ftn  -fPIC -Wall -Wno-unused-variable -Wno-unused-dummy-argument -O   ${FOPTFLAGS} ${FFLAGS}
-----------------------------------------

Using include paths: -I/lustre/home1/u/cekees/.hashdist/tmp/petsc-l62hty3nzk4z/arch-linux2-c-opt/include -I/lustre/home1/u/cekees/.hashdist/tmp/petsc-l62hty3nzk4z/include -I/lustre/home1/u/cekees/.hashdist/tmp/petsc-l62hty3nzk4z/include -I/lustre/home1/u/cekees/.hashdist/tmp/petsc-l62hty3nzk4z/arch-linux2-c-opt/include -I/lustre/home1/u/cekees/.hashdist/bld/parmetis/asuw7mtvy2yg/include
-----------------------------------------

Using C linker: cc
Using Fortran linker: ftn
Using libraries: -Wl,-rpath,/lustre/home1/u/cekees/.hashdist/tmp/petsc-l62hty3nzk4z/arch-linux2-c-opt/lib -L/lustre/home1/u/cekees/.hashdist/tmp/petsc-l62hty3nzk4z/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/lustre/home1/u/cekees/.hashdist/tmp/petsc-l62hty3nzk4z/arch-linux2-c-opt/lib -L/lustre/home1/u/cekees/.hashdist/tmp/petsc-l62hty3nzk4z/arch-linux2-c-opt/lib -lHYPRE -Wl,-rpath,/opt/cray/udreg/2.3.2-1.0402.7546.1.5.gem/lib64 -L/opt/cray/udreg/2.3.2-1.0402.7546.1.5.gem/lib64 -Wl,-rpath,/opt/cray/ugni/5.0-1.0402.7551.1.10.gem/lib64 -L/opt/cray/ugni/5.0-1.0402.7551.1.10.gem/lib64 -Wl,-rpath,/opt/cray/pmi/5.0.1-1.0000.9799.94.9.gem/lib64 -L/opt/cray/pmi/5.0.1-1.0000.9799.94.9.gem/lib64 -Wl,-rpath,/opt/cray/dmapp/4.0.1-1.0402.7784.4.1.gem/lib64 -L/opt/cray/dmapp/4.0.1-1.0402.7784.4.1.gem/lib64 -Wl,-rpath,/opt/cray/xpmem/0.1-2.0402.45248.1.5.gem/lib64 -L/opt/cray/xpmem/0.1-2.0402.45248.1.5.gem/lib64 -Wl,-rpath,/opt/cray/rca/1.0.0-2.0402.47290.7.1.gem/lib64 -L/opt/cray/rca/1.0.0-2.0402.47290.7.1.gem/lib64 -Wl,-rpath,/opt/cray/mpt/6.0.0/gni/mpich2-gnu/48/lib -L/opt/cray/mpt/6.0.0/gni/mpich2-gnu/48/lib -Wl,-rpath,/opt/acml/5.3.1/gfortran64/lib -L/opt/acml/5.3.1/gfortran64/lib -Wl,-rpath,/opt/cray/xe-sysroot/4.2.34/usr/lib64 -L/opt/cray/xe-sysroot/4.2.34/usr/lib64 -Wl,-rpath,/opt/cray/xe-sysroot/4.2.34/lib64 -L/opt/cray/xe-sysroot/4.2.34/lib64 -Wl,-rpath,/opt/cray/xe-sysroot/4.2.34/usr/lib/alps -L/opt/cray/xe-sysroot/4.2.34/usr/lib/alps -Wl,-rpath,/usr/lib/alps -L/usr/lib/alps -Wl,-rpath,/lustre/home1/opt/gcc/4.8.2/snos/lib/gcc/x86_64-suse-linux/4.8.2 -L/lustre/home1/opt/gcc/4.8.2/snos/lib/gcc/x86_64-suse-linux/4.8.2 -Wl,-rpath,/lustre/home1/opt/gcc/4.8.2/snos/lib/gcc -L/lustre/home1/opt/gcc/4.8.2/snos/lib/gcc -Wl,-rpath,/lustre/home1/opt/gcc/4.8.2/snos/lib64 -L/lustre/home1/opt/gcc/4.8.2/snos/lib64 -Wl,-rpath,/lustre/home1/opt/gcc/4.8.2/snos/lib -L/lustre/home1/opt/gcc/4.8.2/snos/lib -Wl,-rpath,/opt/cray/atp/1.7.1/lib -L/opt/cray/atp/1.7.1/lib -lmpichcxx_gnu_48 -lstdc++ -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_dist_3.3 -lsuperlu_4.3 -lX11 -Wl,-rpath,/lustre/home1/u/cekees/.hashdist/bld/parmetis/asuw7mtvy2yg/lib -L/lustre/home1/u/cekees/.hashdist/bld/parmetis/asuw7mtvy2yg/lib -lparmetis -lmetis -lptesmumps -lptscotch -lptscotcherr -lscotch -lscotcherr -lpthread -lgfortran -lm -lgfortran -lm -lm -lm -lquadmath -lm -lmpichcxx_gnu_48 -lstdc++ -lrt -lm -lz -ldl -lrca -lAtpSigHCommData -lAtpSigHandler -lgfortran -lacml -lrt -lmpich_gnu_48 -lmpichf90_gnu_48 -lmpl -lpmi -lalpslli -lalpsutil -lpthread -lgcc_s -ldl
-----------------------------------------