Repository: gerlero/foamlib

Issue:
Support for non-field binary-encoded lists
Hi @gerlero,

As we previously discussed, it would be a really nice feature if we could parse the binary-encoded versions of non-field lists, as opposed to just as the ASCII versions.

Without this the mesh files like faces, points, neighbours,... are difficult to work with if your project is using binary.

FoamFile (for example this but **binary encoded**):
```
/--------------------------------- C++ -----------------------------------
| o | |
| o o | HELYX(R) Core: Open-source CFD for Enterprise |
| o O o | Version: 4.1.1 |
| o o | ENGYS Ltd. http://engys.com/ |
| o | |
*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format binary;
class regIOobject;
arch "LSB;label=32;scalar=64";
location "constant/polyMesh";
object cellZones;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

1000000
(
1
2
3
4
...
999997
999998
999999
1000000
)
// ************************************************************************* //
```

```
from foamlib import FoamFile

path = 'path_to_foam_file...'
U = FoamFile(path)
print(U[None])
```

```
  File "/Users/Christopher.Braley/Documents/projects/apqx/openfoam/foamlib_constant.py", line 27, in parse_file
    print(U[None])
          ~^^^^^^
  File "/Users/Christopher.Braley/Documents/projects/apqx/openfoam/venv/lib/python3.11/site-packages/foamlib/_files/_files.py", line 246, in __getitem__
    parsed = self._get_parsed()
             ^^^^^^^^^^^^^^^^^^
  File "/Users/Christopher.Braley/Documents/projects/apqx/openfoam/venv/lib/python3.11/site-packages/foamlib/_files/_io.py", line 66, in _get_parsed
    self.__parsed = Parsed(contents)
                    ^^^^^^^^^^^^^^^^
  File "/Users/Christopher.Braley/Documents/projects/apqx/openfoam/venv/lib/python3.11/site-packages/foamlib/_files/_parsing.py", line 439, in __init__
    for parse_result in _FILE.parse_string(
                        ^^^^^^^^^^^^^^^^^^^
  File "/Users/Christopher.Braley/Documents/projects/apqx/openfoam/venv/lib/python3.11/site-packages/pyparsing/core.py", line 1219, in parse_string
    raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected end of text, found '('  (at char 841), (line:21, col:1)
```

Many thanks!

Hints from issue comments:
@gerlero: @cjbraley I'll do some work to enable this and we'll see how it works out in practice.@gerlero: Let's list the types in a `polyMesh` first just in case:
 
- `boundary`: `dict` -> always ASCII
- `faces`: `list[4-list[int]]` -> 16 bytes/element in binary (assuming `int32`)
- `neighbour`: `list[int]` -> 4 bytes/element in binary (assuming `int32`)
- `owner`: `list[int]` -> also 4 bytes/element in binary (assuming `int32`)
- `points`: `list[vector] => list[3-list[float]]` -> 24 bytes/element (`double`/`float64`) or 12 bytes/element (`float`/`float32`) in binary

The conclusion from this is that we should be able to autodetect the type of a binary list from its size in bytes and number of elements, without having to rely on other external stuff like the the name of the file.@cjbraley: @gerlero ,

I think that's almost right. The one discrepancy I see is that for us faces can actually have a mixture of 3 & 4 length sub-lists.

It seems like the length of the sub-list is actually arbitrary, so all we know is that it's a **list[list[int]]**. Even **within a single file sub-lists can have different lengths**.

`Each face definition starts with the number of vertices that define that face. For example, 4( indicates a quadrilateral (4-sided) face, and 3( indicates a triangular (3-sided) face.`

Based on my research the types of the sub-lists in the other classes are consistent.

Faces (list of list of int with arbitrary size, but usually 3/4):
```
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v1912                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       faceList;
    location    "constant/polyMesh";
    object      faces;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //


446448146
(
3(904040 904479 924424)
3(904479 904039 924424)
3(904481 904479 924424)
3(904040 904481 924424)
...
4(3517816 3516892 3517751 3516554)
4(3516790 3516787 3516789 3516788)
4(3516980 3516982 3517731 3516981)
4(3516596 3516562 3516560 3516595)
)
```

Neighbour (list of int):
```
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v1912                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       labelList;
    note        "nPoints:69229432  nCells:197914227  nFaces:446448146  nInternalFaces:441573273";
    location    "constant/polyMesh";
    object      neighbour;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //


441573273
(
1
2
3
4
5
6
7
...
197914223
197914219
197914220
197914222
197914223
197914226
)


// ************************************************************************* //
```

Owner (list of int):

```
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v1912                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       labelList;
    note        "nPoints:69229432  nCells:197914227  nFaces:446448146  nInternalFaces:441573273";
    location    "constant/polyMesh";
    object      owner;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //


446448146
(
0
0
1
1
2
2
2
...
138920170
125095048
110302756
110302178
117550292
)


// ************************************************************************* //

```

Points (list of vector):

```
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v1912                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       vectorField;
    location    "constant/polyMesh";
    object      points;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //


69229432
(
(1.9651 3.571886497513121e-18 0.468999999999996)
(1.964838616647891 -0.006931990539489501 0.4689999999999958)
(1.964037752895968 -0.01391940630148043 0.4689999999999963)
(1.962548278381021 -0.02147948475297981 0.4689999999999959)
...
(1.092260254127397 -0.3151451624161979 0.05406851296852164)
(2.461613398668586 -0.5905097235548232 0.0415838960382229)
(0.2032821554166777 -0.1886504205189976 0.3134021318602374)
(0.767235839694649 -0.5245151548925305 0.03938924639997934)
)


// ************************************************************************* //

```



@gerlero: > I think that's almost right. The one discrepancy I see is that for us faces can actually have a mixture of 3 & 4 length sub-lists.
> 
> It seems like the length of the sub-list is actually arbitrary, so all we know is that it's a **list[list[int]]**. Even **within a single file sub-lists can have different lengths**.
> 
> `Each face definition starts with the number of vertices that define that face. For example, 4( indicates a quadrilateral (4-sided) face, and 3( indicates a triangular (3-sided) face.`

Oh interesting, thanks! Unfortunately, this practically rules out the use of NumPy arrays for representation (since arrays cannot be ragged).

I wonder how these are stored when in binary (is the length of each sub-list encoded in the binary stream? I guess it must be).

Anyway, even if we can come up with a fast parser for `list[list[int]]`, representation as a list might make the output unmanageable in terms of memory... not sure what to do with this.

Maybe we could parse it as a [masked array](https://numpy.org/doc/stable/reference/maskedarray.generic.html#constructing-masked-arrays)? (Ideally from the outside it should look like and work just as if were a list of lists, which I'm not sure masked arrays do...)@cjbraley: I see the problem, that makes faces a tough one. Unfortunately, I'm not expert enough with numpy to really guide you here. I will do some research and see if I can find anything. The only way I can think of doing this is basically what you say, use the longest sub-list and put nulls elsewhere.

If it is helpful, I can try and create an example of a binary encoded faces file which includes subarrays of length 3 and 4 for you. 

I did just speak to someone who knows OpenFoam much better than I, and they were under the impression that faces in almost all cases should be triangular. So maybe it makes sense to assume this as a base case for now and leave the multi-format parsing for a future update. Will ask some more questions on this and let you know what I find out.@HenningScheufler: This datastructure could help:

https://exasim-project.com/NeoN/latest/basics/segmentedField.html

You put all value in an continous array and have another array that provided an offset into the first one. This provided the same functionality as a list[list[int]]@gerlero: >If it is helpful, I can try and create an example of a binary encoded faces file which includes subarrays of length 3 and 4 for you.

@cjbraley Please do. A couple of short examples of binary lists along with their expected values would make for great test cases. 

A large test case that pushes the limits of the library might be helpful too.

>This datastructure could help:
>
>https://exasim-project.com/NeoN/latest/basics/segmentedField.html
>
>You put all value in an continous array and have another array that provided an offset into the first one. This provided the same functionality as a list[list[int]]

@HenningScheufler thanks. I'll look into it. I guess it might be worth it if it can be encapsulated in a `Sequence`-based class.@cjbraley: Hi @gerlero 

Hopefully this helps. This was created from the ASCII versions by using the foamFormatConvert command. I don't think I am going to be able to share one of the large files unfortunately.
[dummy_data.zip](https://github.com/user-attachments/files/20128951/dummy_data.zip)

Otherwise, I also had a little more insight from someone who works with the faces data regularly. 

> They are using numpy with two arrays. One to record the start/stop of each face and the other to contain all the points in a flattened list.

That sounds very similar to the suggestion Henning made.

Have a good weekend@gerlero: #376 should allow fast parsing `neighbour`, `owner` and `points` even in binary form (NOTE: I have not tested it, and will probably merge that PR without doing so because it fixes a few bugs with the parser).

Fast parsing of `faces` is still not supported in any format (for ASCII it will fall back to the slow parser; binary will fail).@cjbraley: This is really awesome work @gerlero! The binary files for `neighbour`, `owner` and `points` are running as intended now and extremely fast. ASCII versions were timing out, these finish in a couple of seconds.@gerlero: > dummy_data.zip

Do you have the corresponding ASCII file to the `faces_binary` file. I can't seem to load the contents of the latter as anything meaningful (I'm probably doing something wrong).@cjbraley: @gerlero,

I think this may be my fault for leaving ... in the ASCII files.


```
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v1912                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       faceList;
    location    "constant/polyMesh";
    object      faces;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //


446448146
(
3(904040 904479 924424)
3(904479 904039 924424)
3(904481 904479 924424)
3(904040 904481 924424)
3(55880317 904479 904039)
3(904039 924424 55880317)
3(904479 55880317 924424)
3(924424 55850535 904479)
3(924424 904481 55850535)
3(904481 904479 55850535)
3(65097588 904481 924424)
3(65097588 924424 904040)
3(904040 904481 65097588)
3(904039 55880317 904504)
3(904504 55880317 904479)
3(924425 55880317 904039)
3(55880317 924425 924424)
3(55880317 924424 55850535)
3(55850535 904479 55880317)
3(55850535 65097588 904481)
3(65097588 55850535 924424)
3(904479 55850535 904423)
3(55850535 904481 904423)
3(904040 65097588 924405)
3(924405 65097588 924424)
3(904481 924405 904040)
3(924405 904481 65097588)
3(924425 904504 904039)
3(904504 924425 55880317)
3(904479 904480 55880317)
3(904480 904504 55880317)
3(55880317 924329 924424)
3(924425 924329 55880317)
3(55850535 55015127 924424)
3(55015127 55880317 924424)
3(55880317 55015127 55850535)
3(55850535 904479 904480)
3(55880317 55850535 904480)
3(55850535 65023565 904481)
3(65097588 904481 65023565)
3(55850535 65097588 65023565)
3(55015127 924424 65097588)
3(55015127 65097588 55850535)
3(904480 55850535 904423)
3(904423 55850535 65023565)
3(904423 65023565 904481)
3(65097588 924424 924357)
3(924357 924405 65097588)
3(904041 904481 924405)
3(65097588 924405 904041)
3(904041 904481 65097588)
3(904038 924425 904504)
3(55880317 924385 924425)
3(924385 904504 924425)
3(904504 924385 55880317)
3(55200855 55880317 904480)
3(55880317 55200855 904504)
3(55200855 904480 904504)
3(55015127 55880317 924329)
3(924424 55015127 924329)
3(924329 55880317 924385)
3(55880317 55200856 55850535)
3(55850535 55200856 55015127)
3(55015127 55200856 55880317)
3(55850535 904480 55200856)
3(55200856 904480 55880317)
3(904481 65023565 904458)
3(65097588 904481 904458)
3(65023565 65097588 904458)
3(55850535 65097588 55850548)
3(65023565 55850535 55850548)
3(65097588 65023565 55850548)
3(924357 55015127 924424)
3(65097588 55015127 924357)
3(55850535 55015127 55850548)
3(55850548 55015127 65097588)
3(55200856 55850535 904423)
3(55200856 904423 904480)
3(55200856 55850535 65023565)
3(904423 55200856 65023565)
4(3517816 3516892 3517751 3516554)
4(3516790 3516787 3516789 3516788)
4(3516980 3516982 3517731 3516981)
4(3516596 3516562 3516560 3516595)
4(3517861 3517079 3517080 3517860)
4(3516885 3516889 3517734 3516887)
4(3517852 3516599 3517116 3517115)
4(3517634 3516951 3516953 3516952)
4(3517783 3516880 3516881 3517859)
4(3516995 3516998 3516999 3517854)
4(3516921 3516923 3516925 3516922)
4(3517857 3517833 3517027 3517026)
4(3517796 3517852 3517115 3517795)
4(3517024 3517833 3517857 3517023)
4(3516926 3516929 3516927 3516930)
4(3516553 3516897 3517785 3516555)
4(3517718 3517072 3517719 3517771)
4(3517069 3517718 3517771 3517068)
4(3516783 3516790 3516788 3517752)
4(3516661 3517697 3516664 3516655)
4(3517781 3517059 3517060 3516410)
4(3517770 3516589 3517769 3516635)
4(3517806 3516988 3516989 3517596)
4(3516697 3516592 3516699 3516698)
4(3516692 3516697 3516698 3517707)
4(3516557 3516865 3516866 3516559)
4(3517838 3516600 3517830 3516597)
4(3516957 3516962 3517863 3516961)
4(3517835 3517722 3516886 3517721)
4(3517757 3517054 3516409 3517053)
4(3516906 3516911 3517824 3516908)
4(3517072 3517074 3517073 3517719)
4(3516807 3516811 3516810 3516806)
4(3516672 3516675 3516676 3516670)
4(3516624 3516623 3516625 3517858)
4(3516779 3516781 3516782 3516778)
4(3516777 3516779 3516778 3516771)
4(3517680 3516733 3517728 3517840)
4(3514865 3517680 3517840 3517681)
4(3516958 3516957 3516961 3516956)
4(3517687 3516828 3517688 3516823)
4(3517700 3516441 3516439 3517815)
4(3516593 3516704 3516707 3516700)
4(3516974 3516973 3517844 3516972)
4(3517764 3516974 3516972 3516965)
4(3516616 3516615 3517662 3516614)
4(3517251 3517253 3517252 3517822)
4(3516951 3516958 3516956 3516953)
4(3516925 3516926 3516930 3516922)
4(3516963 3517764 3516965 3516964)
4(3516623 3517772 3517754 3516625)
4(3517399 3517149 3517398 3517793)
4(3517143 3517399 3517793 3517794)
4(3516781 3516783 3517752 3516782)
4(3517252 3517253 3517258 3517259)
4(3517123 3517131 3517135 3517127)
4(3517035 3517747 3517836 3517481)
4(3516600 3516603 3517736 3517830)
4(3517051 3517757 3517053 3517677)
4(3517139 3517143 3517794 3517140)
4(3517384 3517251 3517822 3517723)
4(3516713 3516714 3516718 3516719)
4(3517701 3516749 3516750 3517703)
4(3517734 3516889 3516892 3517816)
4(3517006 3517015 3517013 3517007)
4(3516727 3514866 3514864 3517683)
4(3517235 3517238 3517239 3517303)
4(3517081 3517084 3517085 3517078)
4(3516712 3516714 3516713 3516711)
4(3516905 3517614 3516907 3517613)
4(3517243 3517249 3517247 3517242)
4(3516998 3517001 3516997 3516999)
4(3514866 3514865 3517681 3514864)
4(3516849 3517750 3516851 3516848)
4(3517304 3517298 3517550 3517382)
4(3517228 3517574 3517302 3517755)
4(3516682 3516683 3516684 3516681)
4(3517574 3517235 3517303 3517302)
4(3517229 3517228 3517755 3517227)
4(3517751 3516553 3516555 3516554)
4(3517155 3517314 3517396 3517388)
4(3517305 3517304 3517382 3517306)
4(3516824 3517687 3516823 3516825)
4(3516772 3516777 3516771 3516773)
4(3517299 3517282 3517342 3517281)
4(3517550 3517298 3517299 3517281)
4(3517194 3517195 3517279 3517192)
4(3517135 3517126 3517128 3517127)
4(3517079 3517081 3517078 3517080)
4(3517059 3517061 3517058 3517060)
4(3517043 3516366 3517044 3516365)
4(3516929 3516928 3516931 3516927)
4(3516866 3516865 3517810 3516864)
4(3516760 3516765 3516766 3516761)
4(3516656 3516661 3516655 3516657)
4(3516631 3516634 3516633 3516632)
)


// ************************************************************************* //

```@gerlero: @cjbraley just to check: the list declares `446448146` elements at the top but obviously has much fewer. Is that correct?@cjbraley: Yes sorry, this is me rushing last Friday. The actual files I made got converted to binary in place, so I lost the copies. The number of elements is much fewer, it would be the number you get when counting how many items there are. Here 176@gerlero: Alright, it looks like the first 177 `(u)int32`s in the binary data are indices (just like @HenningScheufler's suggested implementation).

```python
>>> np.frombuffer(contents[contents.find(b"(")+1:contents.find(b"(")+1+177*4], dtype=np.int32)
array([  0,   3,   6,   9,  12,  15,  18,  21,  24,  27,  30,  33,  36,
        39,  42,  45,  48,  51,  54,  57,  60,  63,  66,  69,  72,  75,
        78,  81,  84,  87,  90,  93,  96,  99, 102, 105, 108, 111, 114,
       117, 120, 123, 126, 129, 132, 135, 138, 141, 144, 147, 150, 153,
       156, 159, 162, 165, 168, 171, 174, 177, 180, 183, 186, 189, 192,
       195, 198, 201, 204, 207, 210, 213, 216, 219, 222, 225, 228, 231,
       234, 237, 240, 244, 248, 252, 256, 260, 264, 268, 272, 276, 280,
       284, 288, 292, 296, 300, 304, 308, 312, 316, 320, 324, 328, 332,
       336, 340, 344, 348, 352, 356, 360, 364, 368, 372, 376, 380, 384,
       388, 392, 396, 400, 404, 408, 412, 416, 420, 424, 428, 432, 436,
       440, 444, 448, 452, 456, 460, 464, 468, 472, 476, 480, 484, 488,
       492, 496, 500, 504, 508, 512, 516, 520, 524, 528, 532, 536, 540,
       544, 548, 552, 556, 560, 564, 568, 572, 576, 580, 584, 588, 592,
       596, 600, 604, 608, 612, 616, 620, 624], dtype=int32)
```@gerlero: However, I'm at a loss on how to interpret the rest of the bytes (i.e., I can't seem to get [these numbers](https://github.com/gerlero/foamlib/issues/374#issuecomment-2873466835))@cjbraley: I remade the binary file in case there was something wrong with it. This should definitely match the ASCII above. 
[faces.zip](https://github.com/user-attachments/files/20173872/faces.zip)

It definitely does go from a 176 length list to a 177 length list when you convert, so the underlying structure is changing.@gerlero: > It definitely does go from a 176 length list to a 177 length list when you convert, so the underlying structure is changing.

Makes sense. I guess it adds a final index pointing to the end of the data....@gerlero: Well, I can't make sense of the binary format. I get:

```python
>>> len(contents[contents.find(b"(")+1:contents.rfind(b")")])
3211
```

which is an odd number of bytes!@cjbraley: That is strange. Maybe it will help to break the problem down as small as we possibly can. I've created some more dummy files to try and do that.

- faces_1: a single element of length 3
- faces_2: a single element of length 4
- facts_3: two elements, one length 3, one length 4 

Maybe it's some kind of separator between the two lists that needs to be accounted for.

[faces_data.zip](https://github.com/user-attachments/files/20188977/faces_data.zip)@gerlero: Thanks. I was missing the second list right after the first one.

I wonder if it is useful/worth it to provide a wrapper class (e.g. like we do with `DimensionSet` and `Dimensioned`) or if it's enough to just return the raw lists (i.e. as a tuple of arrays).@cjbraley: @gerlero,

Awesome, sounds like you are making progress!

I think on balance it probably makes sense to do that. It gives you a place to explain what the data struture means and how to use it. Someone who hasn't explored the faces file might be lost otherwise.

On a slightly related note I was actually having some difficulty with the wrapper classes. Specifically that when you compute a dictionary (e.g. loads()), any Dimensioned object that appears somewhere more deeply nested in the structure does not get converted to a native python object. So currently I am doing postprocessing to go through every node in these dicts / lists and turning them in a python list if I find a Dimensioned. 

So similarly, for example if I use .loads() on the faces file, I would want to get the tuple of np.arrays, not the wrapper object.

I wasn't really sure if this qualifies as an issue. Wanted to get your thoughts before raising anything.

In this example, referenceFields.p gets converted to a Dimensioned Object. 

```
/*--------------------------------*- C++ -*----------------------------------*\
|       o          |                                                          |
|    o     o       | HELYX(R) Core: Open-source CFD for Enterprise            |
|   o   O   o      | Version: 4.1.1                                           |
|    o     o       | ENGYS Ltd. <http://engys.com/>                           |
|       o          |                                                          |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      binary;
    class       dictionary;
    arch        "LSB;label=32;scalar=64";
    location    "../system";
    object      materialProperties;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

materialName    air;

materialType    fluid;

referenceFields
{
    p               [ 1 -1 -2 0 0 0 0 ] 100670;
}


// ************************************************************************* //

```

@gerlero: > I think on balance it probably makes sense to do that. It gives you a place to explain what the data struture means and how to use it. Someone who hasn't explored the faces file might be lost otherwise.

Thanks. I'll obviously take this opinion into account.

> On a slightly related note I was actually having some difficulty with the wrapper classes. Specifically that when you compute a dictionary (e.g. loads()), any Dimensioned object that appears somewhere more deeply nested in the structure does not get converted to a native python object. So currently I am doing postprocessing to go through every node in these dicts / lists and turning them in a python list if I find a Dimensioned.
> 
> So similarly, for example if I use .loads() on the faces file, I would want to get the tuple of np.arrays, not the wrapper object.
> 
> I wasn't really sure if this qualifies as an issue. Wanted to get your thoughts before raising anything.

In my mind (and thus in foamlib's design), `Dimensioned` objects are more than just a tuple of `([name ,] dimensions, value)`. In the same line, `DimensionSet` objects are something different than just lists of numbers. That's why the wrapper classes exist for these two—**I didn't see a direct correspondence for these with any of the Python builtin classes nor NumPy arrays.**

The other fact is the design of the `loads` and `dumps` static methods: in my head, these should work as much like the `FoamFile` class itself as possible. I would have to a see very convincing reason to ever deviate from this.

I understand what you're referring too though, so (if I may), let me rephrase the question as: is there anything I could add to `Dimensioned` (and `DimensionSet`) that could make working with these easier for you? E.g., I've recently added support for arithmetic operations to both, which allows for doing some basic math without having to extract a `Dimensioned`'s `value`...@gerlero: @cjbraley #398 adds: 

- A `FoamFile.FaceList` class (implemented as per @HenningScheufler's suggestion) that can stand in for a `list[list[int]]` but should be much more space efficient.
- A fast (regex-based) parser for ASCII `faceLists` (assumes faces are all size 3 or 4 and the size of each is written in the file) that returns a `FaceList`. Note that conversion in the parser will realize the `list[list[int]]` for a moment while constructing the `FaceList`, so I'm not sure how it will perform with very large lists (there may exist a more efficient implementation but I haven't given it any thought). If any of the assumptions fail, the list will still be parsed, but it will be returned as a regular `list[list[int]]`.
- A fast (regex-based) parser for binary `faceCompactLists` that maps 1:1 to `FaceList`, so it should be very efficient.

What's missing:

- Binary serialization of `FaceList`s. I haven't gotten around to implementing this yet.

Seeing that this is a not so minor change, and **especially given that it introduces a new class into the library**, I'd like your opinion on whether you think this way of doing it makes sense for both the library and your particular workflow; and that there are no other more viable alternatives that you can think of.@cjbraley: Thanks @gerlero, really appreciate your efforts here.

That sounds really promising. I definitely think you are on the right track, as I mentioned, I have spoken to some people who work with OpenFoam + Python every day and I was told they use essentially the same two list construct to handle faces data. Definitely support this being a class as it gives a good place to write code that can help interact with the data structure and somewhere to document it.

For my workflow, what I would really want as an output, I think is a tuple of numpy arrays (one that stores all the values, one that stores the offsets of the sublists). Maybe there can be a separate method for turning this into a regular list of lists, but for our use case, that's just not going to be performant enough. Our (binary) faces file alone is 7GB and I would expect other projects to also have very large faces files. Once we can get it into a more workable format (for us parquet), we will probably do the conversion to something that looks more like the ASCII format ourselves.



@gerlero: > For my workflow, what I would really want as an output, I think is a tuple of numpy arrays (one that stores all the values, one that stores the offsets of the sublists). 

Not to take away from your other opinions, but if it works for you without the wrapper class, maybe we can use a different approach without the wrapper class for starters. This would let me push back on the decision on whether the wrapper class is worth it.@cjbraley: That would work for me. Also, please feel welcome to ignore any of my opinions as I don't know the project (or openFoam) nearly as well as you do.@gerlero: > That would work for me. Also, please feel welcome to ignore any of my opinions as I don't know the project (or openFoam) nearly as well as you do.

Can't really develop a useful library without taking into account the opinions of actual users. Thanks again for all the feedback. I'll try to add logic to just return a tuple of arrays for the binary case for now.
@gerlero: @cjbraley Parsing and serialization of all of the mentioned examples should be working as of `0.9.7` (with no wrapper classes)@cjbraley: Thanks @gerlero, very impressed with your output. I am now able to parse the enormous 7GB faces file which is a huge step forward. Takes about 90s on my local machine (MBP M4) which I think is good enough performance wise.



Solution patch:
diff --git a/foamlib/_files/_parsing.py b/foamlib/_files/_parsing.py
index a965a3f..4cc4099 100644
--- a/foamlib/_files/_parsing.py
+++ b/foamlib/_files/_parsing.py
@@ -190,6 +190,60 @@ def _binary_numeric_list(
     ).add_parse_action(to_array)
 
 
+def _ascii_face_list(*, ignore: Regex | None = None) -> ParserElement:
+    element_pattern = r"(?:-?\d+)"
+    spacing_pattern = (
+        rf"(?:(?:\s|{ignore.re.pattern})+)" if ignore is not None else r"(?:\s+)"
+    )
+
+    element_pattern = rf"(?:(?:3{spacing_pattern}?\((?:{element_pattern}{spacing_pattern}){{2}}{element_pattern}{spacing_pattern}?\))|(?:4{spacing_pattern}?\((?:{element_pattern}{spacing_pattern}){{3}}{element_pattern}{spacing_pattern}?\)))"
+
+    list_ = Forward()
+
+    def process_count(tks: ParseResults) -> None:
+        nonlocal list_
+        if not tks:
+            count = None
+        else:
+            (count,) = tks
+            assert isinstance(count, int)
+
+        if count is None:
+            list_pattern = rf"\({spacing_pattern}?(?:{element_pattern}{spacing_pattern})*{element_pattern}{spacing_pattern}?\)"
+
+        elif count == 0:
+            list_ <<= NoMatch()
+            return
+
+        else:
+            list_pattern = rf"\({spacing_pattern}?(?:{element_pattern}{spacing_pattern}){{{count - 1}}}{element_pattern}{spacing_pattern}?\)"
+
+        list_ <<= Regex(list_pattern).add_parse_action(to_face_list)
+
+    def to_face_list(
+        tks: ParseResults,
+    ) -> list[list[np.ndarray[tuple[int], np.dtype[np.int64]]]]:
+        (s,) = tks
+        assert s.startswith("(")
+        assert s.endswith(")")
+        if ignore is not None:
+            s = re.sub(ignore.re, " ", s)
+        s = s.replace("(", " ").replace(")", " ")
+
+        raw = np.fromstring(s, sep=" ", dtype=int)
+
+        values: list[np.ndarray[tuple[int], np.dtype[np.int64]]] = []
+        i = 0
+        while i < raw.size:
+            assert raw[i] in (3, 4)
+            values.append(raw[i + 1 : i + raw[i] + 1])  # type: ignore[arg-type]
+            i += raw[i] + 1
+
+        return [values]
+
+    return Opt(common.integer).add_parse_action(process_count).suppress() + list_
+
+
 def _list_of(entry: ParserElement) -> ParserElement:
     return (
         (
@@ -391,9 +445,14 @@ _DATA <<= _DATA_ENTRY[1, ...].set_parse_action(
 
 _STANDALONE_DATA = (
     _ascii_numeric_list(dtype=int, ignore=_COMMENT)
+    | _ascii_face_list(ignore=_COMMENT)
     | _ascii_numeric_list(dtype=float, nested=3, ignore=_COMMENT)
-    | _binary_numeric_list(dtype=np.int64)
-    | _binary_numeric_list(dtype=np.int32)
+    | (
+        _binary_numeric_list(dtype=np.int64) + Opt(_binary_numeric_list(dtype=np.int64))
+    ).add_parse_action(lambda tks: tuple(tks) if len(tks) > 1 else tks[0])
+    | (
+        _binary_numeric_list(dtype=np.int32) + Opt(_binary_numeric_list(dtype=np.int32))
+    ).add_parse_action(lambda tks: tuple(tks) if len(tks) > 1 else tks[0])
     | _binary_numeric_list(dtype=np.float64, nested=3)
     | _binary_numeric_list(dtype=np.float32, nested=3)
     | _DATA
diff --git a/foamlib/_files/_serialization.py b/foamlib/_files/_serialization.py
index 6464677..132c0c7 100644
--- a/foamlib/_files/_serialization.py
+++ b/foamlib/_files/_serialization.py
@@ -88,7 +88,7 @@ def normalize_data(
                 if arr.ndim == 1 or (arr.ndim == 2 and arr.shape[1] in (3, 6, 9)):
                     return arr  # type: ignore [return-value]
 
-            return [normalize_data(d) for d in data]  # type: ignore [arg-type, misc]
+            return [normalize_data(d) for d in data]  # type: ignore [arg-type, return-value]
 
         if isinstance(data, int):
             return float(data)
@@ -115,7 +115,7 @@ def normalize_data(
         assert not isinstance(k, Mapping)
         return (  # type: ignore [return-value]
             normalize_keyword(k),  # type: ignore [arg-type]
-            normalize_data(v) if not isinstance(v, Mapping) else v,  # type: ignore [arg-type, misc]
+            normalize_data(v) if not isinstance(v, Mapping) else v,  # type: ignore [arg-type]
         )
 
     if (
@@ -123,10 +123,10 @@ def normalize_data(
         and not isinstance(data, DimensionSet)
         and not isinstance(data, tuple)
     ):
-        return [normalize_data(d) for d in data]  # type: ignore [arg-type, misc]
+        return [normalize_data(d) for d in data]  # type: ignore [arg-type, return-value]
 
     if isinstance(data, tuple) and not isinstance(data, DimensionSet):
-        return tuple(normalize_data(d) for d in data)  # type: ignore [misc]
+        return tuple(normalize_data(d, keywords=keywords) for d in data)  # type: ignore [misc]
 
     if isinstance(data, str):
         s = loads(data, keywords=keywords)
@@ -264,7 +264,7 @@ def dumps(
                 ret += b";"
             return ret
 
-        return b" ".join(dumps(v) for v in data)
+        return b" ".join(dumps(v, keywords=keywords, header=header) for v in data)
 
     if is_sequence(data):
         return (
diff --git a/foamlib/_files/_types.py b/foamlib/_files/_types.py
index 8458b64..bf1eab1 100644
--- a/foamlib/_files/_types.py
+++ b/foamlib/_files/_types.py
@@ -234,12 +234,19 @@ DataLike = Union[
 StandaloneData = Union[
     Data,
     "np.ndarray[tuple[int], np.dtype[np.int64 | np.int32]]",
-    "np.ndarray[tuple[int], np.dtype[np.float64 | np.float32]]",
+    "np.ndarray[tuple[int, int], np.dtype[np.float64 | np.float32]]",
+    List["np.ndarray[tuple[int], np.dtype[np.int64 | np.int32]]"],
+    Tuple[
+        "np.ndarray[tuple[int], np.dtype[np.int64 | np.int32]]",
+        "np.ndarray[tuple[int], np.dtype[np.int64 | np.int32]]",
+    ],
 ]
 StandaloneDataLike = Union[
+    StandaloneData,
     DataLike,
-    "np.ndarray[tuple[int], np.dtype[np.int64 | np.int32]]",
-    "np.ndarray[tuple[int], np.dtype[np.float64 | np.float32]]",
+    Sequence["np.ndarray[tuple[int], np.dtype[np.int64 | np.int32]]"],
+    Sequence[Sequence[int]],
+    Tuple[Sequence[int], Sequence[int]],
 ]
 
 


Test patch:
diff --git a/tests/test_files/test_dumps.py b/tests/test_files/test_dumps.py
index 669d635..7589cc5 100644
--- a/tests/test_files/test_dumps.py
+++ b/tests/test_files/test_dumps.py
@@ -129,3 +129,14 @@ def test_serialize_file() -> None:
         FoamFile.dumps({"#include": "$FOAM_CASE/simControls"}, ensure_header=False)
         == b"\n#include $FOAM_CASE/simControls\n"
     )
+    indices = np.array([0, 3, 7], dtype=np.int32)
+    values = np.array(
+        [904040, 904479, 924424, 3516631, 3516634, 3516633, 3516632], dtype=np.int32
+    )
+    assert (
+        FoamFile.dumps(
+            {"FoamFile": {"format": "binary"}, None: (indices, values)},  # type: ignore[dict-item]
+            ensure_header=False,
+        )
+        == b"FoamFile {format binary;} 3(\x00\x00\x00\x00\x03\x00\x00\x00\x07\x00\x00\x00) 7(h\xcb\r\x00\x1f\xcd\r\x00\x08\x1b\x0e\x00\xd7\xa85\x00\xda\xa85\x00\xd9\xa85\x00\xd8\xa85\x00)"
+    )
diff --git a/tests/test_files/test_files.py b/tests/test_files/test_files.py
index bff6709..168f262 100644
--- a/tests/test_files/test_files.py
+++ b/tests/test_files/test_files.py
@@ -91,11 +91,13 @@ def test_write_read(tmp_path: Path) -> None:
     assert sd["y"] == 2
 
     with d:
-        lst = d["subdict", "list"]
+        sd = d["subdict"]
+        assert isinstance(sd, FoamFile.SubDict)
+        lst = sd["list"]
         assert isinstance(lst, list)
         lst[0] = 0
         assert lst == [0, 2, 3]
-        assert d["subdict", "list"] == [1, 2, 3]
+        assert sd["list"] == [1, 2, 3]
 
 
 def test_new_field(tmp_path: Path) -> None:
diff --git a/tests/test_files/test_parsing/test_loads.py b/tests/test_files/test_parsing/test_loads.py
index 459ee18..a501be8 100644
--- a/tests/test_files/test_parsing/test_loads.py
+++ b/tests/test_files/test_parsing/test_loads.py
@@ -74,3 +74,29 @@ def test_loads() -> None:
     assert FoamFile.loads("#include $FOAM_CASE/simControls") == {
         "#include": "$FOAM_CASE/simControls"
     }
+    faces = FoamFile.loads("2(3(1 2 3) 4(4 5 6 7))")
+    assert isinstance(faces, list)
+    assert len(faces) == 2
+    assert isinstance(faces[0], np.ndarray)
+    assert faces[0].shape == (3,)
+    assert faces[0].dtype == np.int64
+    assert isinstance(faces[1], np.ndarray)
+    assert faces[1].shape == (4,)
+    assert faces[1].dtype == np.int64
+    assert np.array_equal(faces[0], [1, 2, 3])
+    assert np.array_equal(faces[1], [4, 5, 6, 7])
+    faces = FoamFile.loads(
+        b"3\n(\x00\x00\x00\x00\x03\x00\x00\x00\x07\x00\x00\x00)\n7\n(h\xcb\r\x00\x1f\xcd\r\x00\x08\x1b\x0e\x00\xd7\xa85\x00\xda\xa85\x00\xd9\xa85\x00\xd8\xa85\x00)"
+    )
+    assert isinstance(faces, tuple)
+    assert len(faces) == 2
+    assert isinstance(faces[0], np.ndarray)
+    assert faces[0].shape == (3,)
+    assert faces[0].dtype == np.int32
+    assert isinstance(faces[1], np.ndarray)
+    assert faces[1].shape == (7,)
+    assert faces[1].dtype == np.int32
+    assert np.array_equal(faces[1][faces[0][0] : faces[0][1]], [904040, 904479, 924424])
+    assert np.array_equal(
+        faces[1][faces[0][1] : faces[0][2]], [3516631, 3516634, 3516633, 3516632]
+    )