opened
joaander
username_0
<!-- Please confirm that your work is based on the correct branch. -->
<!-- Bug fixes should based on *maint*. -->
<!-- New features should based on *master*. -->

## Description

<!-- Describe your changes in detail. -->
Use `gsd.c` from gsd v1.9.2

## Motivation and Context

<!--- Why is this change required? What problem does it solve? -->
This fixes a bug where GSD was unable to write or read data chunks larger than 2 GiB.

## How Has This Been Tested?

<!--- Please describe in detail how you tested your changes. -->
Extensive CI tests in the GSD package.

## Change log

No new change log entry is required.

## Checklist:

- [x] I have reviewed the [**Contributor Guidelines**](https://github.com/glotzerlab/hoomd-blue/blob/master/CONTRIBUTING.md).
- [x] I agree with the terms of the [**HOOMD-blue Contributor Agreement**](https://github.com/glotzerlab/hoomd-blue/blob/master/ContributorAgreement.md).
- [x] My name is on the [list of contributors](https://github.com/glotzerlab/hoomd-blue/blob/master/sphinx-doc/credits.rst).
Bump embedded `gsd.c` to v1.9.2
issue
created
InnocentBug
username_1
I cannot find a method to comment a file not touched by this PR.
But the hoomd/GSDReader.cc should be changed in order to read large files.
```
-- a/hoomd/GSDReader.cc
+++ b/hoomd/GSDReader.cc
@@ -271,7 +271,7 @@ void GSDReader::readHeader()
 */
 void GSDReader::readParticles()
     {
-    unsigned int N = m_snapshot->particle_data.size;
+    size_t N = m_snapshot->particle_data.size;
     m_snapshot->particle_data.type_mapping = readTypes(m_frame, "particles/types");
 
     // the snapshot already has default values, if a chunk is not found, the value
```
The reason is that later `N` is multiplied with integer literals (result is a integer, which may overflow for large `N`).
The overflowing argument is than passed to a function that expects `size_t`.

This is the only position I catched before my tests die (memory exhausted). Might be similar occurrences at other places (also in the Writer?).
comment
created
joaander
username_0
Thanks, I fixed this and similar issues in the writer. I don't have any way of testing that these fixes will work correctly for large N, however.
comment
created
InnocentBug
username_1
I got access to a 1TB memory machine. A test system with 385 mio particles seems to run fine.
At least the configuration is read in and written back to disk and nothing dies. (No actual integration is performed. No potentials but 1 step with an NVE integrator. See script at the bottom.)
The simulation with a single MPI rank and no GPU consumed about 400GB memory.
```python
#!/usr/bin/env python3
import time
import os

import numpy

import hoomd
hoomd.context.initialize()
import hoomd.md as md

def main():
    context = hoomd.context.SimulationContext()
    with context:
        # 385 mio particles, 19GB disk space
        system = hoomd.init.read_gsd("coordS.gsd",time_step=0)

        md.integrate.mode_standard(dt=1e-8)
        integrate_AB = md.integrate.nve(group=hoomd.group.all())

        hoomd.run(1)

        final = hoomd.dump.gsd("coordSF.gsd",group=hoomd.group.all(),period=None,truncate=True,overwrite=True)


if __name__ == "__main__":
    main()
```
comment
created
joaander
username_0
Thanks for testing!
comment