Repository: i18next/i18next (JavaScript)

Issue:
Large numbers of namespaces * languages use higher CPU than necessary - O(n^2)
<!--
Before you submit an issue we recommend you visit [docs](https://www.i18next.com/) or [docs](https://react.i18next.com/) or [StackOverflow](https://stackoverflow.com/search?q=i18next) or similar and ask any questions you have or mention any problems you've had getting started with i18next.

**Please read this entire template before posting any issue. If you ignore these instructions and post an issue here that does not follow the instructions, your issue might be closed, locked.**
-->

## 🐛 Bug Report

`O(n^2)` CPU usage on the number of `namespaces * languages` passed to `BackendConnector.load()`.

On a particular application I work on there are ~20 languages and ~400 namespaces with 8,000 total files.  The application is using `next-i18next` with the `i18next-fs-backend`.  In this case the loading of the files takes ~4 seconds total on a Linux docker image running on Mac / EC2 / Lambda / etc.  We are aiming to reduce this as much as possible and noticed that the loader code is using arrays and `indexOf` calls on those arrays on the total number of `languages * namespaces`.

The fix in #1768 reduces the loading time for 10,000 items to 2% of what it was before and changes the O(n^2) relationship to O(n).

## To Reproduce

- 1,000 items takes `0.083s` on a particular machine
- 10,000 items takes `6.97s` on that same machine
  - 84x longer duration for 10x more input

### 1,000 Items Baseline

<img width="929" alt="image" src="https://user-images.githubusercontent.com/5617868/167280180-db815418-af17-4b4b-ac83-f39329e796de.png">

### 10,000 Items

<img width="923" alt="image" src="https://user-images.githubusercontent.com/5617868/167280184-d230c3ea-0e70-4535-aa11-032ce19da8e0.png">

```js
import BackendConnector from '../../src/BackendConnector.js';
import BackendMock from './backendMock.js';
import Interpolator from '../../src/Interpolator.js';
import ResourceStore from '../../src/ResourceStore.js';
import {expect} from 'chai';

describe('BackendConnector performance test', () => {
  let connector;

  before(() => {
    connector = new BackendConnector(
      new BackendMock(),
      new ResourceStore(),
      {
        interpolator: new Interpolator(),
      },
      {
        backend: { loadPath: 'http://localhost:9876/locales/{{lng}}/{{ns}}.json' },
      },
    );
  });

  describe('#load', () => {
    it('should load 10,000 items in under the 2 second timeout', (done) => {
      const namespaces = [];
      for (let i = 0; i < 10000; i++) {
        namespaces.push(`namespace${i}`);
      }
      connector.load(['en'], namespaces, function (err) {
        expect(err).to.be.not.ok;
        expect(connector.store.getResourceBundle('en', 'namespace1')).to.eql({
          status: 'nok',
          retries: 0,
        });
        done();
      });
    });
  });
});
```

## Expected behavior

- The computation time should grow linearly relative to the `languages * namespaces` count
- Processing `10,000` items should take less time than it does currently

### 1,000 Items Baseline

<img width="989" alt="image" src="https://user-images.githubusercontent.com/5617868/167280402-5720f811-cbaa-4251-97af-a32d329096e1.png">

### 10,000 Items

<img width="1000" alt="image" src="https://user-images.githubusercontent.com/5617868/167280413-386ec77b-874a-4566-9bd2-7786d2696a5a.png">


## Your Environment

- *runtime version*: node v14
- *i18next version*: repo clone on May 7, 2022
- *os*: Mac / Linux
- *any other relevant information*



Discussion:
wow, a lot of namespaces....
[PR](https://github.com/i18next/i18next/pull/1768) looks good to me...
regarding #1769 I've proposed a [less invasive approach](https://github.com/i18next/i18next/issues/1769#issuecomment-1120357392)...
Just out of curiosity...I'm wondering what system needs to load that amount of translations file. Do you have a namespace per component? Do you have to load every language and every application section on init?

I know larger apps that might get to a few hundred files with lazy loading as needed...but this numbers...I have difficulties to see what that system could be using such numbers.

Further how whose numbers of file get translated...looks like a lot of difficulties to overcome...
@jamuhl I am not positive on what drove the namespace count but yes I think it may be one per component. I will see if I can get some of the folks involved in that pattern to share the reasoning.

We load the namespaces on init partly because of the unbounded concurrency and CPU usage from loading large numbers. We noticed that for the first few minutes after startup the app would stall. We moved init to startup in an attempt to limit down delays on requests for a ready instance.

With the CPU usage reduction we may be able to remove the init on startup.

Hmm... now that I think about it I think the O(n^2) problem would have happened 100 times if 100 requests for the same namespaces happened before they finished loading. I need to check that but that could explain the improvement we saw in moving loading to startup. 



Patch:
diff --git a/src/BackendConnector.js b/src/BackendConnector.js
index 40eb6ce30..685536f32 100644
--- a/src/BackendConnector.js
+++ b/src/BackendConnector.js
@@ -2,13 +2,9 @@ import * as utils from './utils.js';
 import baseLogger from './logger.js';
 import EventEmitter from './EventEmitter.js';
 
-function remove(arr, what) {
-  let found = arr.indexOf(what);
-
-  while (found !== -1) {
-    arr.splice(found, 1);
-    found = arr.indexOf(what);
-  }
+function removePending(q, name) {
+  delete q.pending[name];
+  q.pendingCount--;
 }
 
 class Connector extends EventEmitter {
@@ -35,10 +31,10 @@ class Connector extends EventEmitter {
 
   queueLoad(languages, namespaces, options, callback) {
     // find what needs to be loaded
-    const toLoad = [];
-    const pending = [];
-    const toLoadLanguages = [];
-    const toLoadNamespaces = [];
+    const toLoad = {};
+    const pending = {};
+    const toLoadLanguages = {};
+    const toLoadNamespaces = {};
 
     languages.forEach((lng) => {
       let hasAllNamespaces = true;
@@ -51,24 +47,25 @@ class Connector extends EventEmitter {
         } else if (this.state[name] < 0) {
           // nothing to do for err
         } else if (this.state[name] === 1) {
-          if (pending.indexOf(name) < 0) pending.push(name);
+          if (pending[name] !== undefined) pending[name] = true;
         } else {
           this.state[name] = 1; // pending
 
           hasAllNamespaces = false;
 
-          if (pending.indexOf(name) < 0) pending.push(name);
-          if (toLoad.indexOf(name) < 0) toLoad.push(name);
-          if (toLoadNamespaces.indexOf(ns) < 0) toLoadNamespaces.push(ns);
+          pending[name] = true;
+          toLoad[name] = true;
+          toLoadNamespaces[ns] = true;
         }
       });
 
-      if (!hasAllNamespaces) toLoadLanguages.push(lng);
+      if (!hasAllNamespaces) toLoadLanguages[lng] = true;
     });
 
-    if (toLoad.length || pending.length) {
+    if (Object.keys(toLoad).length || Object.keys(pending).length) {
       this.queue.push({
         pending,
+        pendingCount: Object.keys(pending).length,
         loaded: {},
         errors: [],
         callback,
@@ -76,10 +73,10 @@ class Connector extends EventEmitter {
     }
 
     return {
-      toLoad,
-      pending,
-      toLoadLanguages,
-      toLoadNamespaces,
+      toLoad: Object.keys(toLoad),
+      pending: Object.keys(pending),
+      toLoadLanguages: Object.keys(toLoadLanguages),
+      toLoadNamespaces: Object.keys(toLoadNamespaces),
     };
   }
 
@@ -103,17 +100,18 @@ class Connector extends EventEmitter {
     // callback if ready
     this.queue.forEach((q) => {
       utils.pushPath(q.loaded, [lng], ns);
-      remove(q.pending, name);
+      removePending(q, name);
 
       if (err) q.errors.push(err);
 
-      if (q.pending.length === 0 && !q.done) {
+      if (q.pendingCount === 0 && !q.done) {
         // only do once per loaded -> this.emit('loaded', q.loaded);
         Object.keys(q.loaded).forEach((l) => {
-          if (!loaded[l]) loaded[l] = [];
-          if (q.loaded[l].length) {
-            q.loaded[l].forEach((ns) => {
-              if (loaded[l].indexOf(ns) < 0) loaded[l].push(ns);
+          if (!loaded[l]) loaded[l] = {};
+          const loadedKeys = Object.keys(loaded[l]);
+          if (loadedKeys.length) {
+            loadedKeys.forEach((ns) => {
+              if (loadedKeys[ns] !== undefined) loaded[l][ns] = true;
             });
           }
         });


Test patch:
diff --git a/test/backend/backendConnector.perf.spec.js b/test/backend/backendConnector.perf.spec.js
new file mode 100644
index 000000000..fc3ee4a1c
--- /dev/null
+++ b/test/backend/backendConnector.perf.spec.js
@@ -0,0 +1,39 @@
+import BackendConnector from '../../src/BackendConnector.js';
+import BackendMock from './backendMock.js';
+import Interpolator from '../../src/Interpolator.js';
+import ResourceStore from '../../src/ResourceStore.js';
+import { expect } from 'chai';
+
+describe('BackendConnector performance test', () => {
+  let connector;
+
+  before(() => {
+    connector = new BackendConnector(
+      new BackendMock(),
+      new ResourceStore(),
+      {
+        interpolator: new Interpolator(),
+      },
+      {
+        backend: { loadPath: 'http://localhost:9876/locales/{{lng}}/{{ns}}.json' },
+      },
+    );
+  });
+
+  describe('#load', () => {
+    it('should load 10,000 items in under the 2 second timeout', (done) => {
+      const namespaces = [];
+      for (let i = 0; i < 10000; i++) {
+        namespaces.push(`namespace${i}`);
+      }
+      connector.load(['en'], namespaces, function (err) {
+        expect(err).to.be.not.ok;
+        expect(connector.store.getResourceBundle('en', 'namespace1')).to.eql({
+          status: 'nok',
+          retries: 0,
+        });
+        done();
+      });
+    });
+  });
+});