Repository: metafacture/metafacture-blog
Pull request #41: Add "fixing memory leak"
Author: login: dr0i
Created: 2025-03-17T14:15:17Z | Merged: 2025-03-20T13:57:33Z

This is a blog post about fixing the memory leak, see metafacture/metafacture-core#666.


Diff:
diff --git a/content/blog/2025-03-10-fixing-memory-leak/index.md b/content/blog/2025-03-10-fixing-memory-leak/index.md
new file mode 100644
index 0000000..d3aad1f
--- /dev/null
+++ b/content/blog/2025-03-10-fixing-memory-leak/index.md
@@ -0,0 +1,82 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimately in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+## Preconditions
+
+Affected were all usages of Metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest since March 2021 when we
+changed how to start our ETL processes for lobid-resources: Back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitoring tool installed on our servers which checks for terminated processes
+and restarts them automatically. This happened often after some
+more ETL processes were invoked (the "daily updates").
+It was unclear why these crashes appeared but by assigning more RAM and 
+(automatically) restarting the whole JVM after crashes the ETL process went
+stable enough.
+
+## Plotting the leak
+
+Since 2025 we started the [Rheinland-Pfälzische Bibliografie (RPB)](https://github.com/hbz/rpb/).
+Here, ETL processes are triggered _every time_ a record is changed or created.
+This resulted very quickly in an
+[OutOfMemoryError](https://github.com/hbz/lobid-resources/issues/2121#issuecomment-2631355294).
+
+Using Intellij IDEA we can easily produce a picture which shows the ever
+increasing demand of memory:
+![Screenshot of the graph showing the increasing demand of memory](./intellijIdeaGraphShowingMemoryLeak.png)
+At the beginning (at the left) we see that the CPU (green colored) is heavily used
+for all classes are compiled and loaded. Then we see how the Heap Memory (blue
+colored) is filled and after some time there are some spikes, indicating that
+the Garbage Collector was able to free some unused memory.
+We can see how the memory
+consumption is ascending nonetheless. Then a long line appears, lacking all
+spikes. This happens when there is less and less memory which the
+garbage collector could release. We can see 5 peaks where the CPU heavily
+works - this is the garbage collector trying to find some piece of memory
+to free - without much avail. The app crashes shortly after that (not to be
+    seen).
+
+## One fix, several problems solved
+
+After [fixing the memory leak](https://github.com/metafacture/metafacture-core/commit/b32609307f75187a6a3822b8a951429c7fc924f3)
+the consumption of memory is normal, i.e. not ascending:
+![Screenshot of the graph showing normal consumption of memory](./intellijIdeaGraphShowingMemoryWithoutLeak.png)
+Every spike indicates that memory resources were freed, resulting in a stable
+rise and fall of CPU and memory usage. The memory leak is really fixed. 😀
+
+Fixing the memory leak in Metafacture resolved some issues we've experienced:
+- lobid-resources: daily updates sometimes aborted - although this was not such a big thing because our monitor scripts could "heal" the update process automatically (by restarting the app). However, the updates now don't take e.g. 4h (counting from triggering the update until the successful ETL) but 0.45m, which is way faster.
+- Metafacture Playground: we had some [performance issues](https://github.com/metafacture/metafacture-playground/issues/194) which are now solved.
+- RPB: a situation arose where we could only ever add more memory to our VMs to counteract a crash of cataloguing - always fearing that not too much documents were ETLed before the daily restart of the cataloguing app.
+
+## How to and credits
+
+It is _one_ thing to discover a memory leak, but another thing to
+determine where the source of that leak _exactly_ is.
+I have to thank e.g. [Chris Braithwaite for his excellent blog post concerning Java memory leaks](https://medium.com/@chrisbrat_17048/java-memory-leak-investigation-8add1314e33b) to gain a bit more of the background of what a Java memory leak is.
+Very useful for me was the built-in profiler in Intellij IDEA. It not only
+has helped to plot the graphs (see above) to see at a glance that there indeed is a
+memory leak, but can also capture memory snapshots and profile the CPU usage
+to find the problematic classes. It would show something like this:
+![Screenshot of the graph showing normal consumption of memory](./intellijIdeaProfilingProcesses.png)
+If you have found the class where the memory leak most likely originates from
+you can make an educated guess of the statement which causes the problem,
+or/and by commenting in/out the likely code, tracking down the problematic code.
diff --git a/content/blog/2025-03-10-fixing-memory-leak/intellijIdeaGraphShowingMemoryLeak.png b/content/blog/2025-03-10-fixing-memory-leak/intellijIdeaGraphShowingMemoryLeak.png
new file mode 100644
index 0000000..f052260
Binary files /dev/null and b/content/blog/2025-03-10-fixing-memory-leak/intellijIdeaGraphShowingMemoryLeak.png differ
diff --git a/content/blog/2025-03-10-fixing-memory-leak/intellijIdeaGraphShowingMemoryWithoutLeak.png b/content/blog/2025-03-10-fixing-memory-leak/intellijIdeaGraphShowingMemoryWithoutLeak.png
new file mode 100644
index 0000000..0b2399e
Binary files /dev/null and b/content/blog/2025-03-10-fixing-memory-leak/intellijIdeaGraphShowingMemoryWithoutLeak.png differ
diff --git a/content/blog/2025-03-10-fixing-memory-leak/intellijIdeaProfilingProcesses.png b/content/blog/2025-03-10-fixing-memory-leak/intellijIdeaProfilingProcesses.png
new file mode 100644
index 0000000..6fab182
Binary files /dev/null and b/content/blog/2025-03-10-fixing-memory-leak/intellijIdeaProfilingProcesses.png differ


Reviews:
user: login: TobiasNx
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T15:02:13Z
user: login: blackwinter
body: Fixed some obvious typos/grammar mistakes.

The notation of "Metafacture" is unclear to me. When referring to the repository specifically it's okay to format it as code, but when referring to the project/framework as a whole I think it should be properly capitalized and not formatted as code.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-17T15:20:17Z
user: login: acka47
body: +1
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T10:01:35Z

Review comments:
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
used uncloseable resources, which would leak memory, [resulting ultimately in a
```
created_at: 2025-03-17T14:59:10Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
experience the leaking of memory at the earliest since March 2021 when we
```
created_at: 2025-03-17T15:03:47Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
+changed how to start our `lobid` ETL processes: back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitor tool installed at our servers who checks if processes
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
We also have a monitor tool installed on our servers which checks if processes
```
created_at: 2025-03-17T15:04:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
+changed how to start our `lobid` ETL processes: back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitor tool installed at our servers who checks if processes
+are terminated and restart those automatically. This happened often after some
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
are terminated and restarts those automatically. This happened often after some
```
created_at: 2025-03-17T15:04:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
+changed how to start our `lobid` ETL processes: back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitor tool installed at our servers who checks if processes
+are terminated and restart those automatically. This happened often after some
+more ETL processes were invoked (the "daily updates").
+It was unclear why these crashes appear but by assigning more RAM and 
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
It was unclear why these crashes appeared but by assigning more RAM and 
```
created_at: 2025-03-17T15:05:25Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
+changed how to start our `lobid` ETL processes: back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitor tool installed at our servers who checks if processes
+are terminated and restart those automatically. This happened often after some
+more ETL processes were invoked (the "daily updates").
+It was unclear why these crashes appear but by assigning more RAM and 
+(automatically) restarting the whole JVM after crashes the ETL process went
+stable enough.
+
+Since 2025 we started the [Rheinland-Pfälzische Bibliografie (RPB)](https://github.com/hbz/rpb/).
+Here, ETL processes are triggered _every time_ a record is changed or created.
+This resulted very fast in an
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
This resulted very quickly in an
```
created_at: 2025-03-17T15:06:49Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
+changed how to start our `lobid` ETL processes: back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitor tool installed at our servers who checks if processes
+are terminated and restart those automatically. This happened often after some
+more ETL processes were invoked (the "daily updates").
+It was unclear why these crashes appear but by assigning more RAM and 
+(automatically) restarting the whole JVM after crashes the ETL process went
+stable enough.
+
+Since 2025 we started the [Rheinland-Pfälzische Bibliografie (RPB)](https://github.com/hbz/rpb/).
+Here, ETL processes are triggered _every time_ a record is changed or created.
+This resulted very fast in an
+[OutOfMemoryError](https://github.com/hbz/lobid-resources/issues/2121#issuecomment-2631355294).
+
+Using Intellij IDEA we can easily produce a picture which shows the ever
+increasing demand of memory:
+![Screenshot of the graph showing the increasing demand of memory](./intellijIdeaGraphShowingMemoryLeak.png)
+At the beginning (at the left) we see that the CPU (green colored) is heavily used
+for all classes are compiled and loaded. Then we see how the Heap Memory (blue
+colored) is filled and after some time there are some spikes, indicating that
+the Garbage Collector was able to free some unused memory.
+We can see how the memory
+consumption is ascending nonetheless. Then a long line appears, lacking all
+spikes. This happens when there is less and less memory which the
+garbage collector could release. We can see 5 peaks where the CPU heavily
+works - this is the garbage collector trying to find some piece of memory
+to free - without much avail. The app crashes shortly after that (not to be
+    seen).
+
+After [fixing the memory leak](https://github.com/metafacture/metafacture-core/commit/b32609307f75187a6a3822b8a951429c7fc924f3)
+the consumption of memory is normal, i.e. not ascending:
+![Screenshot of the graph showing normal consumption of memory](./intellijIdeaGraphShowingMemoryWithoutLeak.png)
+Every spike indicates that memory resources were freed, resulting in a stable
+rise and fall of CPU and memory usage. The memory leak is really fixed. 😀
+
+Fixing the memory leak in `metafacture` resolved some issues we've experienced
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
Fixing the memory leak in `metafacture` resolved some issues we've experienced:
```
created_at: 2025-03-17T15:09:45Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
+changed how to start our `lobid` ETL processes: back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitor tool installed at our servers who checks if processes
+are terminated and restart those automatically. This happened often after some
+more ETL processes were invoked (the "daily updates").
+It was unclear why these crashes appear but by assigning more RAM and 
+(automatically) restarting the whole JVM after crashes the ETL process went
+stable enough.
+
+Since 2025 we started the [Rheinland-Pfälzische Bibliografie (RPB)](https://github.com/hbz/rpb/).
+Here, ETL processes are triggered _every time_ a record is changed or created.
+This resulted very fast in an
+[OutOfMemoryError](https://github.com/hbz/lobid-resources/issues/2121#issuecomment-2631355294).
+
+Using Intellij IDEA we can easily produce a picture which shows the ever
+increasing demand of memory:
+![Screenshot of the graph showing the increasing demand of memory](./intellijIdeaGraphShowingMemoryLeak.png)
+At the beginning (at the left) we see that the CPU (green colored) is heavily used
+for all classes are compiled and loaded. Then we see how the Heap Memory (blue
+colored) is filled and after some time there are some spikes, indicating that
+the Garbage Collector was able to free some unused memory.
+We can see how the memory
+consumption is ascending nonetheless. Then a long line appears, lacking all
+spikes. This happens when there is less and less memory which the
+garbage collector could release. We can see 5 peaks where the CPU heavily
+works - this is the garbage collector trying to find some piece of memory
+to free - without much avail. The app crashes shortly after that (not to be
+    seen).
+
+After [fixing the memory leak](https://github.com/metafacture/metafacture-core/commit/b32609307f75187a6a3822b8a951429c7fc924f3)
+the consumption of memory is normal, i.e. not ascending:
+![Screenshot of the graph showing normal consumption of memory](./intellijIdeaGraphShowingMemoryWithoutLeak.png)
+Every spike indicates that memory resources were freed, resulting in a stable
+rise and fall of CPU and memory usage. The memory leak is really fixed. 😀
+
+Fixing the memory leak in `metafacture` resolved some issues we've experienced
+at:
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
```
created_at: 2025-03-17T15:09:52Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
+changed how to start our `lobid` ETL processes: back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitor tool installed at our servers who checks if processes
+are terminated and restart those automatically. This happened often after some
+more ETL processes were invoked (the "daily updates").
+It was unclear why these crashes appear but by assigning more RAM and 
+(automatically) restarting the whole JVM after crashes the ETL process went
+stable enough.
+
+Since 2025 we started the [Rheinland-Pfälzische Bibliografie (RPB)](https://github.com/hbz/rpb/).
+Here, ETL processes are triggered _every time_ a record is changed or created.
+This resulted very fast in an
+[OutOfMemoryError](https://github.com/hbz/lobid-resources/issues/2121#issuecomment-2631355294).
+
+Using Intellij IDEA we can easily produce a picture which shows the ever
+increasing demand of memory:
+![Screenshot of the graph showing the increasing demand of memory](./intellijIdeaGraphShowingMemoryLeak.png)
+At the beginning (at the left) we see that the CPU (green colored) is heavily used
+for all classes are compiled and loaded. Then we see how the Heap Memory (blue
+colored) is filled and after some time there are some spikes, indicating that
+the Garbage Collector was able to free some unused memory.
+We can see how the memory
+consumption is ascending nonetheless. Then a long line appears, lacking all
+spikes. This happens when there is less and less memory which the
+garbage collector could release. We can see 5 peaks where the CPU heavily
+works - this is the garbage collector trying to find some piece of memory
+to free - without much avail. The app crashes shortly after that (not to be
+    seen).
+
+After [fixing the memory leak](https://github.com/metafacture/metafacture-core/commit/b32609307f75187a6a3822b8a951429c7fc924f3)
+the consumption of memory is normal, i.e. not ascending:
+![Screenshot of the graph showing normal consumption of memory](./intellijIdeaGraphShowingMemoryWithoutLeak.png)
+Every spike indicates that memory resources were freed, resulting in a stable
+rise and fall of CPU and memory usage. The memory leak is really fixed. 😀
+
+Fixing the memory leak in `metafacture` resolved some issues we've experienced
+at:
+- `lobid-resources`: daily updates sometimes aborted - although this was not so a big thing because our monitor scripts could "heal" the update process automatically (by restarting the app). However, the updates now don't take e.g. 4h (counting from triggering the update until the successful ETL) but 0.45m, which is way faster.
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
- `lobid-resources`: daily updates sometimes aborted - although this was not such a big thing because our monitor scripts could "heal" the update process automatically (by restarting the app). However, the updates now don't take e.g. 4h (counting from triggering the update until the successful ETL) but 0.45m, which is way faster.
```
created_at: 2025-03-17T15:10:14Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
+changed how to start our `lobid` ETL processes: back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitor tool installed at our servers who checks if processes
+are terminated and restart those automatically. This happened often after some
+more ETL processes were invoked (the "daily updates").
+It was unclear why these crashes appear but by assigning more RAM and 
+(automatically) restarting the whole JVM after crashes the ETL process went
+stable enough.
+
+Since 2025 we started the [Rheinland-Pfälzische Bibliografie (RPB)](https://github.com/hbz/rpb/).
+Here, ETL processes are triggered _every time_ a record is changed or created.
+This resulted very fast in an
+[OutOfMemoryError](https://github.com/hbz/lobid-resources/issues/2121#issuecomment-2631355294).
+
+Using Intellij IDEA we can easily produce a picture which shows the ever
+increasing demand of memory:
+![Screenshot of the graph showing the increasing demand of memory](./intellijIdeaGraphShowingMemoryLeak.png)
+At the beginning (at the left) we see that the CPU (green colored) is heavily used
+for all classes are compiled and loaded. Then we see how the Heap Memory (blue
+colored) is filled and after some time there are some spikes, indicating that
+the Garbage Collector was able to free some unused memory.
+We can see how the memory
+consumption is ascending nonetheless. Then a long line appears, lacking all
+spikes. This happens when there is less and less memory which the
+garbage collector could release. We can see 5 peaks where the CPU heavily
+works - this is the garbage collector trying to find some piece of memory
+to free - without much avail. The app crashes shortly after that (not to be
+    seen).
+
+After [fixing the memory leak](https://github.com/metafacture/metafacture-core/commit/b32609307f75187a6a3822b8a951429c7fc924f3)
+the consumption of memory is normal, i.e. not ascending:
+![Screenshot of the graph showing normal consumption of memory](./intellijIdeaGraphShowingMemoryWithoutLeak.png)
+Every spike indicates that memory resources were freed, resulting in a stable
+rise and fall of CPU and memory usage. The memory leak is really fixed. 😀
+
+Fixing the memory leak in `metafacture` resolved some issues we've experienced
+at:
+- `lobid-resources`: daily updates sometimes aborted - although this was not so a big thing because our monitor scripts could "heal" the update process automatically (by restarting the app). However, the updates now don't take e.g. 4h (counting from triggering the update until the successful ETL) but 0.45m, which is way faster.
+- `metafacture Playground`: we had some [performance issues](https://github.com/metafacture/metafacture-playground/issues/194) which are now solved
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
- `Metafacture Playground`: we had some [performance issues](https://github.com/metafacture/metafacture-playground/issues/194) which are now solved.
```
created_at: 2025-03-17T15:11:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
+changed how to start our `lobid` ETL processes: back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitor tool installed at our servers who checks if processes
+are terminated and restart those automatically. This happened often after some
+more ETL processes were invoked (the "daily updates").
+It was unclear why these crashes appear but by assigning more RAM and 
+(automatically) restarting the whole JVM after crashes the ETL process went
+stable enough.
+
+Since 2025 we started the [Rheinland-Pfälzische Bibliografie (RPB)](https://github.com/hbz/rpb/).
+Here, ETL processes are triggered _every time_ a record is changed or created.
+This resulted very fast in an
+[OutOfMemoryError](https://github.com/hbz/lobid-resources/issues/2121#issuecomment-2631355294).
+
+Using Intellij IDEA we can easily produce a picture which shows the ever
+increasing demand of memory:
+![Screenshot of the graph showing the increasing demand of memory](./intellijIdeaGraphShowingMemoryLeak.png)
+At the beginning (at the left) we see that the CPU (green colored) is heavily used
+for all classes are compiled and loaded. Then we see how the Heap Memory (blue
+colored) is filled and after some time there are some spikes, indicating that
+the Garbage Collector was able to free some unused memory.
+We can see how the memory
+consumption is ascending nonetheless. Then a long line appears, lacking all
+spikes. This happens when there is less and less memory which the
+garbage collector could release. We can see 5 peaks where the CPU heavily
+works - this is the garbage collector trying to find some piece of memory
+to free - without much avail. The app crashes shortly after that (not to be
+    seen).
+
+After [fixing the memory leak](https://github.com/metafacture/metafacture-core/commit/b32609307f75187a6a3822b8a951429c7fc924f3)
+the consumption of memory is normal, i.e. not ascending:
+![Screenshot of the graph showing normal consumption of memory](./intellijIdeaGraphShowingMemoryWithoutLeak.png)
+Every spike indicates that memory resources were freed, resulting in a stable
+rise and fall of CPU and memory usage. The memory leak is really fixed. 😀
+
+Fixing the memory leak in `metafacture` resolved some issues we've experienced
+at:
+- `lobid-resources`: daily updates sometimes aborted - although this was not so a big thing because our monitor scripts could "heal" the update process automatically (by restarting the app). However, the updates now don't take e.g. 4h (counting from triggering the update until the successful ETL) but 0.45m, which is way faster.
+- `metafacture Playground`: we had some [performance issues](https://github.com/metafacture/metafacture-playground/issues/194) which are now solved
+- `RPB`: a situation arised where we could only ever add more memory to our VMs to counteract a crash of cataloguing - always fearing that not too much documents were ETLed before the daily restart of the cataloguing app.
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
- `RPB`: a situation arose where we could only ever add more memory to our VMs to counteract a crash of cataloguing - always fearing that not too much documents were ETLed before the daily restart of the cataloguing app.
```
created_at: 2025-03-17T15:11:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+---
+title: Fixing a memory leak
+description: "How we were affected by a memory leak and how we fixed it"
+date: "2025-03-17"
+authors: [{lastname: "Christoph",
+           firstname: "Pascal"}]
+---
+
+Some `metafacture-core` classes (e.g. the ubiquitous [`FileMap`](https://github.com/metafacture/metafacture-core/blob/master/metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java) (which is invoked when loading a file as a table (like a CSV) in your ETL scripts)
+used uncloseable resources, which would leak memory, [resulting ultimatley in a
+"OutOfMemoryError: Java heap space"](https://github.com/metafacture/metafacture-core/issues/666) since at least 2013 (back then the class was
+ called `MapFile`).
+
+Affected were all usages of metafacture which instantiate a `Flux` multiple times
+over the lifecycle of one JVM. While this is an obvious statement, we could
+experience the leaking of memory at the earliest of March 2021 as we
+changed how to start our `lobid` ETL processes: back in the early days we invoked
+the ETL by starting a new JVM, run our workflow and terminate the JVM afterwards.
+Coming with the [Webhook in March 2021](https://github.com/hbz/lobid-resources/issues/1159)
+the JVM was not terminated after an ETL but listened further for incoming ETL
+triggering. In principle the JVM should have only been restarted when new code
+was to be deployed, but we soon discovered that restarting [our Play app](https://github.com/hbz/lobid-resources/tree/master/web)
+just before the weekly fresh and complete ETL of > 24 million documents improved
+the performance and averted some hanging processes or crashes.
+
+We also have a monitor tool installed at our servers who checks if processes
+are terminated and restart those automatically. This happened often after some
+more ETL processes were invoked (the "daily updates").
+It was unclear why these crashes appear but by assigning more RAM and 
+(automatically) restarting the whole JVM after crashes the ETL process went
+stable enough.
+
+Since 2025 we started the [Rheinland-Pfälzische Bibliografie (RPB)](https://github.com/hbz/rpb/).
+Here, ETL processes are triggered _every time_ a record is changed or created.
+This resulted very fast in an
+[OutOfMemoryError](https://github.com/hbz/lobid-resources/issues/2121#issuecomment-2631355294).
+
+Using Intellij IDEA we can easily produce a picture which shows the ever
+increasing demand of memory:
+![Screenshot of the graph showing the increasing demand of memory](./intellijIdeaGraphShowingMemoryLeak.png)
+At the beginning (at the left) we see that the CPU (green colored) is heavily used
+for all classes are compiled and loaded. Then we see how the Heap Memory (blue
+colored) is filled and after some time there are some spikes, indicating that
+the Garbage Collector was able to free some unused memory.
+We can see how the memory
+consumption is ascending nonetheless. Then a long line appears, lacking all
+spikes. This happens when there is less and less memory which the
+garbage collector could release. We can see 5 peaks where the CPU heavily
+works - this is the garbage collector trying to find some piece of memory
+to free - without much avail. The app crashes shortly after that (not to be
+    seen).
+
+After [fixing the memory leak](https://github.com/metafacture/metafacture-core/commit/b32609307f75187a6a3822b8a951429c7fc924f3)
+the consumption of memory is normal, i.e. not ascending:
+![Screenshot of the graph showing normal consumption of memory](./intellijIdeaGraphShowingMemoryWithoutLeak.png)
+Every spike indicates that memory resources were freed, resulting in a stable
+rise and fall of CPU and memory usage. The memory leak is really fixed. 😀
+
+Fixing the memory leak in `metafacture` resolved some issues we've experienced
+at:
+- `lobid-resources`: daily updates sometimes aborted - although this was not so a big thing because our monitor scripts could "heal" the update process automatically (by restarting the app). However, the updates now don't take e.g. 4h (counting from triggering the update until the successful ETL) but 0.45m, which is way faster.
+- `metafacture Playground`: we had some [performance issues](https://github.com/metafacture/metafacture-playground/issues/194) which are now solved
+- `RPB`: a situation arised where we could only ever add more memory to our VMs to counteract a crash of cataloguing - always fearing that not too much documents were ETLed before the daily restart of the cataloguing app.
+
+# How to and credits
+It is _one_ thing to discover a memory leak, but another thing to
+determine where the source of that leak _exactly_ is.
+I've to thank e.g. [Chris Braithwaite for his excellent blog post concerning Java memory leaks](https://medium.com/@chrisbrat_17048/java-memory-leak-investigation-8add1314e33b) to gain a bit more of the background of what a Java memory leak is.
+Very useful for me was the build-in profiler in Intellij IDEA. It not only
path: content/blog/2025-03-10-fixing-memory-leak/index.md
user: login: blackwinter
body: ```suggestion
Very useful for me was the built-in profiler in Intellij IDEA. It not only
```
created_at: 2025-03-17T15:12:22Z
author_association: MEMBER

PR comments:
user: login: dr0i
created_at: 2025-03-17T14:17:10Z
body: After merging we should announce the post at https://metadaten.community/c/software-und-tools/metafacture/8.
author_association: MEMBER
user: login: TobiasNx
created_at: 2025-03-17T15:00:52Z
body: +1 looks good
author_association: CONTRIBUTOR
user: login: dr0i
created_at: 2025-03-17T15:37:19Z
body: Thanks for the fixes!
author_association: MEMBER
user: login: acka47
created_at: 2025-03-18T10:01:25Z
body: I added some structure/headings, removed some code markup and adjusted some phrasing. +1 from me.
author_association: CONTRIBUTOR