Repository: mikro-orm/mikro-orm (TypeScript)

Issue:
Dataloader failure with ManyToMany relationship
### Describe the bug

With a simple many to many relationship on an entity like this:

```
  @ManyToMany(() => Country)
  countries = new Collection<Country>(this);
```

 Enabling dataloader to load this collection with e.g. `await user.countries.load({dataloader: true})` fails with the following error:

```
TypeError: Cannot read properties of undefined (reading '__helper')

      at helper (node_modules/@mikro-orm/core/entity/wrap.js:24:19)
      at node_modules/@mikro-orm/core/utils/DataloaderUtils.js:155:42
          at Array.filter (<anonymous>)
      at node_modules/@mikro-orm/core/utils/DataloaderUtils.js:179:33
          at Array.map (<anonymous>)
      at DataLoader._batchLoadFn (node_modules/@mikro-orm/core/utils/DataloaderUtils.js:171:34)
```

### Reproduction

https://github.com/tackley/mikro-orm-reproduction

test that illustrates the issue in https://github.com/tackley/mikro-orm-reproduction/blob/master/src/dataloader-m-to-m.test.ts

### What driver are you using?

@mikro-orm/postgresql

### MikroORM version

6.3.10

### Node.js version

20.17.0

### Operating system

Linux

### Validations

- [X] Read the [Contributing Guidelines](https://github.com/mikro-orm/mikro-orm/blob/master/CONTRIBUTING.md).
- [X] Read the [docs](https://mikro-orm.io/docs).
- [X] Check that there isn't [already an issue](https://github.com/mikro-orm/mikro-orm/issues) that reports the same bug to avoid creating a duplicate.
- [X] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/mikro-orm/mikro-orm/discussions) or join our [Discord](https://discord.gg/w8bjxFHS7X).
- [X] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.
Issue with dataloader and many-to-many relations
### Describe the bug

When initializing a collection defined through a many-to-many relation (f.ex. `product.categories.init({ dataloader: true })`), using the dataloader functionality can cause a major performance issue. The generated SQL query with dataloader used differs from a corresponding non-dataloader query so that it adds an unneeded extra join into the query, causing potentially a huge amount of duplicated, unneeded rows appearing in the SQL execution result. In my case I noticed that doing something like `product.categories.init({ dataloader: true })` could take 30 seconds, while `product.categories.init({ dataloader: false })` would take a couple hundred milliseconds.

Here are the two (formatted) SQL queries the above two examples would generate. First, with dataloader:

```sql
select
	`c0` .*,
	`p1`.`id` as `p1__id`
from
	`category` as `c0`
left join `category_products` as `c2` on
	`c0`.`id` = `c2`.`category_id`
left join `product` as `p1` on
	`c2`.`product_id` = `p1`.`id`
left join `category_products` as `c3` on
	`c0`.`id` = `c3`.`category_id`
where
	`c3`.`product_id` in (61876)
```

And then without dataloader:	

```sql
select
	`c1` .*,
	`c0`.`product_id` as `fk__product_id`,
	`c0`.`category_id` as `fk__category_id`
from
	`category_products` as `c0`
inner join `category` as `c1` on
	`c0`.`category_id` = `c1`.`id`
where
	`c0`.`product_id` in (61876)
```

As you can see there's one more join in the dataloader example. In this case it's the pivot table `category_products` being joined twice, and the double join of the table causes unneeded / duplicated extra rows that satisfy the where condition appearing in the end result (potentially so many extra rows that they would cause a major performance issue). In the above example, the dataloader-enabled query would work if the `c3` join wasn't done and `c2.product_id` was used in the where condition instead. I included more info and samples in the reproduction repo.

This behavior is surprising since one would expect using the dataloader to just mostly improve performance, and it can sneakily cause issues if somebody has dataloader enabled globally. I didn't manage to find the issue in the source code, but it looks like this could be remedied by altering the SQL MikroORM generates, to not introduce double joins when dataloader is used.

### Reproduction

https://github.com/sosmo/mikroorm-reproduction

### What driver are you using?

@mikro-orm/mysql

### MikroORM version

6.4.10

### Node.js version

22

### Operating system

Linux

### Validations

- [x] Read the [Contributing Guidelines](https://github.com/mikro-orm/mikro-orm/blob/master/CONTRIBUTING.md).
- [x] Read the [docs](https://mikro-orm.io/docs).
- [x] Check that there isn't [already an issue](https://github.com/mikro-orm/mikro-orm/issues) that reports the same bug to avoid creating a duplicate.
- [x] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/mikro-orm/mikro-orm/discussions) or join our [Discord](https://discord.gg/w8bjxFHS7X).
- [x] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.
Collection init with dataloader not supporting ref: true
### Describe the bug

It seems like if I do something like `product.categories.init({ dataloader: true, ref: true })` (where product is a MikroORM entity and `categories` is a many-to-many collection), I'll get an error with the message "Entity 'Category' does not have property ':ref'". Could this be something I'm doing wrong, or is it a bug? A concrete example is in the reproduction github repo as the second test case.

### Reproduction

https://github.com/sosmo/mikroorm-reproduction

### What driver are you using?

@mikro-orm/mysql

### MikroORM version

6.4.10

### Node.js version

22

### Operating system

Linux

### Validations

- [x] Read the [Contributing Guidelines](https://github.com/mikro-orm/mikro-orm/blob/master/CONTRIBUTING.md).
- [x] Read the [docs](https://mikro-orm.io/docs).
- [x] Check that there isn't [already an issue](https://github.com/mikro-orm/mikro-orm/issues) that reports the same bug to avoid creating a duplicate.
- [x] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/mikro-orm/mikro-orm/discussions) or join our [Discord](https://discord.gg/w8bjxFHS7X).
- [x] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.


Discussion: The problem is specifically a unidirectional M:N relation, if you add the inverse side it works fine.

@darkbasic could you look into this please?
> The problem is specifically a unidirectional M:N relation, if you add the inverse side it works fine

I think that's exactly what's going on. I will further look into this but I'm pretty sure that I need an inverse side in order to compute the relationship and be able to reassign the results.
This needs to be documented at the very least but I'm wondering if it could be possible to dynamically prohibit dataloader usage whenever the M:N relationship lacks an inverse side.
Uh, why? I don't see how an inverse side should be a limiting factor for this. This is something to fix, not do document.
I took a closer look, and the whole implementation of the dataloader filters feels quite duplicated, the EntityLoader already implements all those things, in the correct way (without the need for inverse sides). Namely the M:N collections are handled here:

https://github.com/mikro-orm/mikro-orm/blob/master/packages/core/src/entity/EntityLoader.ts#L458

I tried to hack it and it seems to be working fine: #6053 (@darkbasic review welcome, I wonder why didn't you try to reuse that when implementing the dataloader support?)

The existing tests are quite sparse, and when looking at the implementation, it feels like it cannot really work correctly with the various options, namely I can't see any place where we would actually try to handle different `where` conditions or child `populate` hints, it feels like the options are internally used only for the keys and not really for what they are meant for. With that said, I don't even see how we could be using them, maybe we should validate and disallow them with the dataloader enabled.
> I took a closer look, and the whole implementation of the dataloader filters feels quite duplicated, the EntityLoader already implements all those things, in the correct way (without the need for inverse sides). I wonder why didn't you try to reuse that when implementing the dataloader support?)

It was born as an out of tree implementation and it gone through several major ORM versions which broke the implementation each time. I don't remember exactly but I guess that's the reason why I ended up using the least amount of internal interfaces subject to breaking changes.

> The existing tests are quite sparse

Yeah even though I noticed that the tests were originally built considering MANY-to-MANY relationships with both inverse and no inverse side: https://github.com/mikro-orm/mikro-orm/blob/master/tests/features/dataloader.test.ts#L49
Yet somehow only the collection with the inverse side ended up being actually tested: https://github.com/mikro-orm/mikro-orm/blob/master/tests/features/dataloader.test.ts#L250
At this point I'm not sure if I forgot to implement the no inverse side case or what else.

> looking at the implementation, it feels like it cannot really work correctly with the various options, namely I can't see any place where we would actually try to handle different where conditions or child populate hints, it feels like the options are internally used only for the keys and not really for what they are meant for

That's expected, the initial mainlined implementation was supposed to handle just the simplest use cases. Of course the typings should be updated to reflect that.

> With that said, I don't even see how we could be using them, maybe we should validate and disallow them with the dataloader enabled.

Handling of `where` and `populate` options has already been implemented in the out of tree library, but it's doesn't support the full set of the operators so it is typed accordingly.
Currently the out of tree dataloader also tries to optimize as many queries/where options combinations into the smallest number of queries possible: that's the main reason why implementing some operators proved to be so tricky.

The first step should be to update the typings to reflect the current capabilities.
Then I would like to merge the where/populate handling from the out of tree module **without** the query optimizer: that way a different query for each where/populate combination should be fired which is still good enough for most use cases. It was already in the plan, but life got in the way and never managed to do so.

> I tried to hack it and it seems to be working fine: https://github.com/mikro-orm/mikro-orm/pull/6053 (@darkbasic review welcome

Sure I will have a look, thanks for doing so.







Fix (patch):
diff --git a/packages/core/src/EntityManager.ts b/packages/core/src/EntityManager.ts
index 6388ed516581..0ed9253bdecb 100644
--- a/packages/core/src/EntityManager.ts
+++ b/packages/core/src/EntityManager.ts
@@ -1959,6 +1959,13 @@ export class EntityManager<Driver extends IDatabaseDriver = IDatabaseDriver> {
     return this.getContext().entityFactory;
   }
 
+  /**
+   * @internal use `em.populate()` as the user facing API, this is exposed only for internal usage
+   */
+  getEntityLoader(): EntityLoader {
+    return this.getContext().entityLoader;
+  }
+
   /**
    * Gets the Hydrator used by the EntityManager.
    */
diff --git a/packages/core/src/entity/Collection.ts b/packages/core/src/entity/Collection.ts
index b4ee8a5eee38..17e7e0a4dd9a 100644
--- a/packages/core/src/entity/Collection.ts
+++ b/packages/core/src/entity/Collection.ts
@@ -15,10 +15,10 @@ import type {
 import { ArrayCollection } from './ArrayCollection';
 import { DataloaderUtils, Utils } from '../utils';
 import { ValidationError } from '../errors';
-import { type QueryOrderMap, ReferenceKind, DataloaderType } from '../enums';
+import { DataloaderType, type QueryOrderMap, ReferenceKind } from '../enums';
 import { Reference } from './Reference';
 import type { Transaction } from '../connections/Connection';
-import type { FindOptions, CountOptions, LoadHint } from '../drivers/IDatabaseDriver';
+import type { CountOptions, FindOptions, LoadHint } from '../drivers/IDatabaseDriver';
 import { helper } from './wrap';
 import type { EntityLoaderOptions } from './EntityLoader';
 
@@ -307,8 +307,15 @@ export class Collection<T extends object, O extends object = object> extends Arr
         { ...options, orderBy },
       ]);
 
-      if (!customOrder) {
-        this.reorderItems(items, order);
+      if (this.property.kind === ReferenceKind.MANY_TO_MANY) {
+        this.initialized = true;
+        this.dirty = false;
+
+        if (!customOrder) {
+          this.reorderItems(items, order);
+        }
+
+        return this as unknown as LoadedCollection<Loaded<TT, P>>;
       }
 
       this.items.clear();
diff --git a/packages/core/src/entity/EntityLoader.ts b/packages/core/src/entity/EntityLoader.ts
index 6a822b41bffe..670af42fcb2c 100644
--- a/packages/core/src/entity/EntityLoader.ts
+++ b/packages/core/src/entity/EntityLoader.ts
@@ -219,7 +219,8 @@ export class EntityLoader {
       .flatMap(orderBy => orderBy[prop.name]);
 
     if (prop.kind === ReferenceKind.MANY_TO_MANY && this.driver.getPlatform().usesPivotTable()) {
-      return this.findChildrenFromPivotTable<Entity>(filtered, prop, options, innerOrderBy as QueryOrderMap<Entity>[], populate, !!ref);
+      const res = await this.findChildrenFromPivotTable<Entity>(filtered, prop, options, innerOrderBy as QueryOrderMap<Entity>[], populate, !!ref);
+      return Utils.flatten(res);
     }
 
     const where = await this.extractChildCondition(options, prop);
@@ -468,7 +469,8 @@ export class EntityLoader {
     });
   }
 
-  private async findChildrenFromPivotTable<Entity extends object>(filtered: Entity[], prop: EntityProperty<Entity>, options: Required<EntityLoaderOptions<Entity>>, orderBy?: QueryOrderMap<Entity>[], populate?: PopulateOptions<Entity>, pivotJoin?: boolean): Promise<AnyEntity[]> {
+  /** @internal */
+  async findChildrenFromPivotTable<Entity extends object>(filtered: Entity[], prop: EntityProperty<Entity>, options: Required<EntityLoaderOptions<Entity>>, orderBy?: QueryOrderMap<Entity>[], populate?: PopulateOptions<Entity>, pivotJoin?: boolean): Promise<AnyEntity[][]> {
     const ids = (filtered as AnyEntity[]).map(e => e.__helper!.__primaryKeys);
     const refresh = options.refresh;
     let where = await this.extractChildCondition(options, prop, true);
@@ -490,7 +492,7 @@ export class EntityLoader {
     }
 
     const map = await this.driver.loadFromPivotTable<any, any>(prop, ids, where, orderBy, this.em.getTransactionContext(), options2, pivotJoin);
-    const children: AnyEntity[] = [];
+    const children: AnyEntity[][] = [];
 
     for (const entity of (filtered as AnyEntity[])) {
       const items = map[entity.__helper!.getSerializedPrimaryKey()].map(item => {
@@ -510,7 +512,7 @@ export class EntityLoader {
         return this.em.getUnitOfWork().register(entity as AnyEntity, item, { refresh, loaded: true });
       });
       (entity[prop.name] as unknown as Collection<AnyEntity>).hydrate(items, true);
-      children.push(...items);
+      children.push(items);
     }
 
     return children;
diff --git a/packages/core/src/utils/DataloaderUtils.ts b/packages/core/src/utils/DataloaderUtils.ts
index bbb3b820273d..717950eb3446 100644
--- a/packages/core/src/utils/DataloaderUtils.ts
+++ b/packages/core/src/utils/DataloaderUtils.ts
@@ -1,10 +1,11 @@
-import type { Primary, Ref } from '../typings';
+import type { Dictionary, Primary, Ref } from '../typings';
 import { Collection, type InitCollectionOptions } from '../entity/Collection';
 import { helper } from '../entity/wrap';
 import { type EntityManager } from '../EntityManager';
 import type DataLoader from 'dataloader';
 import { DataloaderType, ReferenceKind } from '../enums';
 import { type LoadReferenceOptions, Reference } from '../entity/Reference';
+import { Utils } from './Utils';
 
 export class DataloaderUtils {
 
@@ -138,7 +139,6 @@ export class DataloaderUtils {
           // We need to populate the inverse side of the relationship in order to be able to later retrieve the PK(s) from its item(s)
           populate: [
             ...(opts.populate === false ? [] : opts.populate ?? []),
-            ...(opts.ref ? [':ref'] : []),
             ...Array.from(filterMap.keys()).filter(
               // We need to do so only if the inverse side is a collection, because we can already retrieve the PK from a reference without having to load it
               prop => em.getMetadata<any>(className).properties[prop]?.ref !== true,
@@ -172,11 +172,6 @@ export class DataloaderUtils {
         return target === collection.owner;
       }
 
-      // FIXME https://github.com/mikro-orm/mikro-orm/issues/6031
-      if (!target && collection.property.kind === ReferenceKind.MANY_TO_MANY) {
-        throw new Error(`Inverse side is required for M:N relations with dataloader: ${collection.owner.constructor.name}.${collection.property.name}`);
-      }
-
       return false;
     };
   }
@@ -187,6 +182,30 @@ export class DataloaderUtils {
    */
   static getColBatchLoadFn(em: EntityManager): DataLoader.BatchLoadFn<[Collection<any>, Omit<InitCollectionOptions<any, any>, 'dataloader'>?], any> {
     return async (collsWithOpts: readonly [Collection<any>, Omit<InitCollectionOptions<any, any>, 'dataloader'>?][]) => {
+      const prop = collsWithOpts[0][0].property;
+
+      if (prop.kind === ReferenceKind.MANY_TO_MANY && em.getPlatform().usesPivotTable()) {
+        const options = {} as Dictionary;
+        const wrap = (cond: unknown) => ({ [prop.name]: cond });
+        const orderBy = Utils.asArray(collsWithOpts[0][1]?.orderBy).map(o => wrap(o));
+        const populate = wrap(collsWithOpts[0][1]?.populate);
+        const owners = collsWithOpts.map(c => c[0].owner);
+        const $or: Dictionary[] = [];
+
+        // a bit of a hack, but we need to prefix the key, since we have only a column name, not a property name
+        const alias = em.config.getNamingStrategy().aliasName(prop.pivotEntity, 0);
+        const fk = `${alias}.${Utils.getPrimaryKeyHash(prop.joinColumns)}`;
+
+        for (const c of collsWithOpts) {
+          $or.push({ $and: [c[1]?.where ?? {}, { [fk]: c[0].owner }] });
+          options.refresh ??= c[1]?.refresh;
+        }
+
+        options.where = wrap({ $or });
+
+        return em.getEntityLoader().findChildrenFromPivotTable(owners, prop, options as any, orderBy as any, populate as any, collsWithOpts[0][1]?.ref);
+      }
+
       const entitiesAndOptsMap = DataloaderUtils.groupInversedOrMappedKeysByEntityAndOpts(collsWithOpts);
       const promises = DataloaderUtils.entitiesAndOptsMapToQueries(entitiesAndOptsMap, em);
       const resultsMap = new Map(await Promise.all(promises));