libMesh/libmesh — pull request #190 by friedmud (2014-01-22 05:50:05+00:00)
Title: use current_local_solution for build_solution_vector

This is not a complete solution, yet... but I wanted to start a discussion.

As I've mentioned before EquationSystems::build_solution_vector() is a PIG... and is now keeping us from running some of our larger problems (anything over 100 Million DoFs takes an enormous amount of memory during build_solution_vector()).

There are multiple problems, but this patch addresses an easy one. There is a vector called sys_soln that gets created in build_solution_vector. The current behavior is to _serialize_ the solution vector of each system to _all_ processors before starting to build the nodal solution vector that ultimately gets passed to an output class.

So, if you have a system with 100 Million DoFs this is creating a vector that is nearly 1GB in size ON EVERY PROCESSOR. So if you are trying to use 32 MPI per node on your cluster... your memory usage _per node_ just went up by nearly 32GB! Considering our cluster has just 64GB of RAM per node... your job most likely just crashed...

So.... to get around this, this patch just uses the current_local_solution vector that we already have hanging around and is properly spread out in parallel and everything. The only weirdness is that it turns out that current_local_solution might not be up to date after the solve so you need to call system.update()... but this is a "const" function... so some const_cast()ing needs to happen. It's kinda ugly, but that's why I want to start a conversation.

What other ways can we solve this issue?

While we're thinking about that start thinking about the next problem which is the number_of_nodes*number_of_vars size vector that we create on every processor in this same function. It has the very same problem... but in this case it's even worse because if you're using a serial outputer (like Exodus) you really only need that vector on processor 0... but we allocate it on every processor! Crazy!