Repository: scverse/scanpy

Issue:
Leiden now included in python-igraph
The Leiden algorithm is now [included](https://igraph.org/python/doc/igraph.Graph-class.html#community_leiden) in the latest release of `python-igraph`, version 0.8.0. I believe this alleviates the need to depend on the `leidenalg` packages. The Leiden algorithm provided in `python-igraph` is substantially faster than the `leidenalg` package. It is simpler though, providing fewer options, but I believe the more extensive options of the `leidenalg` package are not necessarily needed for the purposes of `scanpy`. We  provide binary wheels on PyPI and binaries for conda are available from the conda-forge channel, also for Windows.


Discussion:
Thanks for the update! I'm not sure if we'll be able to migrate very easily though. We allow users to [choose the quality function](https://scanpy.readthedocs.io/en/stable/api/scanpy.tl.leiden.html#scanpy.tl.leiden), and use the `leidenalg.RBConfigurationVertexPartition` as the default. We've also been considering using the multiplex partitioning methods.

* Do you think the performance improvements will also be implemented in leidenalg?
* Is modularity with a resolution parameter equivalent to `leidenalg.RBConfigurationVertexPartition`?
The Leiden algorithm from `igraph` only implements two quality functions: CPM and modularity. Both indeed use resolution parameters, and are the equivalent of `CPMVertexPartition` and `RBConfigurationVertexPartition`. The reason the `igraph` implementation is more performant than `leidenalg` is exactly because it does not provide other quality functions or a multiplex approach. If you need the other quality functions (such as significance or surprise), you will need to stick to `leidenalg`. So if you need those other quality functions, or a multiplex approach, you might still want to stick to the `leidenalg` package after all.
If we can cleanly switch to the igraph implementation for modularity with weights, it could make sense for that to be the default. Any chance you could point me to some benchmarks on performance? An initial test looks very impressive!
I haven't performed an in-depth benchmark comparison. But results from a single run of modularity detection on an example (a [Facebook graph](http://konect.uni-koblenz.de/networks/facebook-wosn-links)) is sufficiently revealing I think:
```
Running Leiden 0.7.0.post1+71.g14ba1e4.dirty
Running igraph 0.8.0
Read graph (n=63731,m=817035), starting community detection.
leidenalg: t=8.048258741036989, m=0.6175825273363675
igraph community_leiden: t=1.159165252931416, m=0.6298702028415605
```
This is only a relatively small graph, and the difference is likely to be even bigger for larger graphs. Perhaps the `igraph` Leiden algorithm can indeed be the default, with `leidenalg` being an optional choice or something?
I have a large single cell set, where i'd love to speed up the Leiden clustering. 

Is it the igraph version of leiden implemented in scanpy? Just wondering if this might be a potential area for speed gains. 

Thanks!
This hasn't been implemented yet, but a pull request would be welcome.

There would also have to be documentation about changing results and how to get previous behavior. Some benchmarks of performance would also be great.
So there are definite speed gains to be had with the igraph implementation of Leiden clustering. But the results are not exactly the same.

I have run igraph straight from adata by piggy backing scanpy utility functions with the following code...
```
adjacency = sc._utils._choose_graph(adata, obsp=None, neighbors_key=None)
g = sc._utils.get_igraph_from_adjacency(adjacency)
clustering = g.community_leiden(objective_function='modularity', resolution_parameter=0.15)
adata_analyse.obs['leiden_igraph'] = pd.Series(clustering.membership, dtype='category', index=adata_analyse.obs.index)
```

On my spatial data (only mention this to explain the poor cell-cell separation in the results!) igraph is 5.86x faster at clustering a 185,000 cell dataset vs. `sc.tl.leiden(adata, resolution=0.15)`. The methods output different cluster numbers, but the results broadly make sense for both outputs biologically speaking. 
![image](https://user-images.githubusercontent.com/25825809/153923576-499b85c3-f521-460e-88d7-be99a4bf055b.png)

On the [PBMC 3k dataset from scanpy tutorials,](https://scanpy-tutorials.readthedocs.io/en/latest/pbmc3k.html#Clustering-the-neighborhood-graph) there is no practically speed difference (too small a dateset to matter). The cluster number output from both methods is the same for both methods. There are a few differences in cell cluster assignment though.
![image](https://user-images.githubusercontent.com/25825809/153925230-1f0becf1-e024-4968-bf3a-410144acecb7.png)



> The Leiden algorithm is now [included](https://igraph.org/python/doc/igraph.Graph-class.html#community_leiden) in the latest release of `python-igraph`, version 0.8.0. I believe this alleviates the need to depend on the `leidenalg` packages. The Leiden algorithm provided in `python-igraph` is substantially faster than the `leidenalg` package. It is simpler though, providing fewer options, but I believe the more extensive options of the `leidenalg` package are not necessarily needed for the purposes of `scanpy`. We provide binary wheels on PyPI and binaries for conda are available from the conda-forge channel, also for Windows.

I have now done a speed comparison with adata object of 1.85 million cells.

igraph on adata as implemented [above](https://github.com/theislab/scanpy/issues/1053#issuecomment-1039424473) ran in **33 minutes** vs `sc.tl.leiden()` which took **~14 hours**


@vtraag @ivirshup I accidentally ran the `igraph_community_leiden` in a for loop, 4 times with the same settings and noticed the leiden output differed slightly each time. Is there a parameter to set to ensure the same output each time? randomseed maybe?

More generally, any advice on which settings to implement with the `igraph_community_leiden` call to replicate `sc.tl.leiden`? It is not clear to me what some of the `sc.tl.leiden` default calls are to `leidenalg`.

Thanks!

Thanks!
Good to see the large speed gains! Do note that `g.community_leiden(objective_function='modularity', resolution_parameter=0.15)` won't use any edge weights by default. Possibly, `sc.tl.leiden()`, might use edge weights? This might explain some of the discrepancies you might see. Another difference is that `community_leiden` only runs on undirected graphs, while `leidenalg` will also work on directed graphs, which makes a difference when using modularity. This could perhaps also explain some of the discrepancies.
> I accidentally ran the `igraph_community_leiden` in a for loop, 4 times with the same settings and noticed the leiden output differed slightly each time. Is there a parameter to set to ensure the same output each time? randomseed maybe?

Yes, you can set a random seed. You can set the RNG in `igraph` using [`set_random_number_generator`](https://igraph.org/python/api/latest/igraph._igraph.html#set_random_number_generator). When simply passing the standard `random` Python library, you can set the seed using [`random.seed`](https://docs.python.org/3/library/random.html#random.seed).
I'm not sure the objective should be to get the same output from two runs. I recall louvain having a low average NMI (below 0.3 or sth like that?) for multiple runs at different random seeds. In the end this is an NP-hard problem with a good heuristic solution that is affected by the random ordering of nodes. Maybe it would be more useful to look at multiple runs of `sc.tl.leiden` and multiple runs of leiden from `python-igraph` and compare the outputs in terms of NMI to one another or to some baseline label assignment?
Getting identical output from two runs without using the same seed is unlikely in many cases indeed. In some cases, you really do want to get identical output, so then setting a seed could be useful.

In addition, it would be good to ensure that at least the approach is conceptually the same for both the `python-igraph` implementation and `sc.tl.leiden`. But the suggestion of taking multiple runs and compare them makes sense!
@vtraag, could you comment on what about this implementation makes it faster? I'm wondering how much the speed gains could be from number of iterations.

Some other notes:

* The typical "connectivity graph" we get from UMAP is symmetric, so not being directed shouldn't be an issue
* We do use weights by default, which seems to give more stable results (especially when number of neighbors changes)
The `leidenalg` package was originally built for flexibility, and you can easily plugin new quality functions. As a result, some of the admin stuff is being done less efficiently than it could be done. In `igraph`, there is less flexibility, so that the implementation can be made more efficient.

Additionally, some of the iteration over neighbours in the `leidenalg` package is less efficient than how it is implemented in `igraph` at the moment. This could be made more efficient though, but it is something that requires quite some rewriting, for which I would first need to find the time. I'm not sure how large the speed gains of this would be immediately.
For the 3k test dataset, introducing edge weights with
```
import random
random.seed(1234) 

adjacency = sc._utils._choose_graph(adata, obsp=None, neighbors_key=None)
g = sc._utils.get_igraph_from_adjacency(adjacency)
clustering = g.community_leiden(objective_function='modularity', weights='weight')
adata.obs['leiden_igraph_edge_weighted'] = pd.Series(clustering.membership, dtype='category', index=adata.obs.index)
```
Leads to a more similar clustering to `sc.tl.leiden`.

Setting the seed makes all reproducible.
![image](https://user-images.githubusercontent.com/25825809/154090475-9b5afd35-f254-4c30-92d5-c1a1a86d797d.png)

Including igraph edge weights does not seem to impact run times on my larger datasets vs. igraph without weights.



I suspect a lot of the runtime increase may come from the number of iterations. 

Using the default value, `n_interations=2`, `g.community_leiden("modularity", weights="weight")` took ~2.5 minutes.

Using `n_iterations=-1`, which is what scanpy does, `g.community_leiden("modularity", weights="weight", n_iterations=-1)` took ~1.5 hours.

I am unsure why we set `n_iterations=-1`, since that's not what `leidenalg` does by default. I'm not seeing much discussion of why it's done this way in the issue (#350) or PR (#361)...
Ah right, you meant those iterations, of course. Yes, that definitely can make a large difference! When comparing the speed of both implementations, indeed the number of iterations should of course be identical.
> @vtraag @ivirshup I accidentally ran the `igraph_community_leiden` in a for loop, 4 times with the same settings and noticed the leiden output differed slightly each time. Is there a parameter to set to ensure the same output each time? randomseed maybe?
> 
> 
> 
> More generally, any advice on which settings to implement with the `igraph_community_leiden` call to replicate `sc.tl.leiden`? It is not clear to me what some of the `sc.tl.leiden` default calls are to `leidenalg`.
> 
> 
> 
> Thanks!
> 
> 
> 
> Thanks!

As mentioned below, you should set the RNG. AFAIK scanpy does that by default.
It would probably good to see how important the `n_iterations` parameter is for our data. Hopefully after the first couple iterations it's only a few points of the million shuffling around per iteration. I'll try to take a closer when I can, but basically need to try something like:

```python
def iterativley_cluster(
    g: igraph.Graph,
    *,
    n_iterations: int = 10,
    random_state: int = 0,
    leiden_kwargs: dict = {}
) -> list:
    import random
    random.seed(random_state)

    _leiden_kwargs = {"objective_function": "modularity", "weights": "weight"}
    _leiden_kwargs.update(leiden_kwargs)

    partition = g.community_leiden(n_iterations=1, **_leiden_kwargs)

    steps = [partition]
    for _ in range(n_iterations-1):
        partition = g.community_leiden(n_iterations=1, initial_membership=partition.membership, **_leiden_kwargs)
        steps.append(partition)

    return steps
```

My suspicion (and hope) would be that unstable clusters / points are the ones that drag on the optimization process. E.g. groups that aren't maintained when you change the random seed also aren't maintained through later iterations.
Setting `n_iterations=-1` in `g.community_leiden` certainly impacts run time (vs. default `n_iterations=2`), making runtimes more similar to `sc.tl.leiden()`. For large datasets though, run times with `g.comunity_leiden` still appear faster.

The average of 4 leiden runs on my 185,000 cell subsampled dataset:
`sc.tl.leiden`, 11.5 minutes
`g.community_leiden`, 9.5 minutes 


1 leiden run on my 1,850,000 cell subsampled dataset:
`sc.tl.leiden`, 11 hours, 26 minutes  
`g.community_leiden`, 7 hours, 30 minutes  



@mezwick, I don't fully understand something. The timings you report seem to be lower for `sc.tl.leiden` than for `g.community_leiden`, suggesting that `g.community_leiden` is *slower* than `sc.tl.leiden`, but in your comments you say that `g.community_leiden` appears faster. Am I misunderstanding something? Could you perhaps clarify?
Sorry, i made a critical typo in the time reports, where i listed the functions the wrong way round. I have updated the comment to correct this. 

To be clear. `g.community_leiden` is faster than `sc.tl.leiden` in my case, particulalrly for large datasets.


> Setting `n_iterations=-1` in `g.community_leiden` certainly impacts run time (vs. default `n_iterations=2`), making runtimes more similar to `sc.tl.leiden()`. For large datasets though, run times with `g.comunity_leiden` still appear faster.
> 
> The average of 4 leiden runs on my 185,000 cell subsampled dataset: `sc.tl.leiden`, 11.5 minutes `g.community_leiden`, 9.5 minutes
> 
> 1 leiden run on my 1,850,000 cell subsampled dataset: `sc.tl.leiden`, 11 hours, 26 minutes `g.community_leiden`, 7 hours, 30 minutes


Hi @vtraag @ivirshup,

I have been unable to get the leiden clustering to run on a large 18.5 million cell dataset with either `sc.tl.leiden()` or the `python-igraph` implementation outlined above. Smaller subsets of the data run fine. I think the problem lies somewhere with weights. 

When i run 
```
obs_name = f"leiden {leiden_suffix}"
sc.tl.leiden(adata, resolution=0.1, n_iterations=2, key_added=obs_name)
```
I get the following traceback
```
Traceback (most recent call last):
  File "/opt/notebooks/output/../scripts/sc_cluster.py", line 92, in <module>
    adjacency = sc._utils._choose_graph(adata, obsp=None, neighbors_key=None)
  File "/opt/conda/envs/scanpy_py3pt9/lib/python3.9/site-packages/scanpy/_utils/__init__.py", line 219, in get_igraph_from_adjacency
    g.es['weight'] = weights
SystemError: /opt/conda/conda-bld/python-split_1649141344976/work/Objects/listobject.c:138: bad argument to internal function
```

When i run the `python-igraph` leiden implementation like so...
```
obs_name = f"leiden {leiden_suffix}"
g = sc._utils.get_igraph_from_adjacency(adjacency)
clustering = g.community_leiden(
    objective_function="modularity", 
    resolution_parameter=0.1,            
    weights = 'weight',
    n_iterations=2
    )
adata.obs[obs_name] = (
    pd.Series(
        clustering.membership, 
        dtype='category', 
        index=adata.obs.index
        )
    )
```
I get the following, similar traceback
```
UMAP leiden clustering resolution 0.1 commenced 12:27:06.619473
running Leiden clustering
Traceback (most recent call last):
  File "/opt/notebooks/output/../scripts/sc_cluster.py", line 88, in <module>
    obs_name = f"leiden {leiden_suffix}"
  File "/opt/conda/envs/scanpy_py3pt9/lib/python3.9/site-packages/scanpy/tools/_leiden.py", line 129, in leiden
    g = _utils.get_igraph_from_adjacency(adjacency, directed=directed)
  File "/opt/conda/envs/scanpy_py3pt9/lib/python3.9/site-packages/scanpy/_utils/__init__.py", line 219, in get_igraph_from_adjacency
    g.es['weight'] = weights
SystemError: /opt/conda/conda-bld/python-split_1649141344976/work/Objects/listobject.c:138: bad argument to internal function

```

The script runs completely fine when I subsample the adata with `sc.pp.neighbours`. So far, i have managed to run 0.5 fraction  of the cells (9.25 million cells). On the cluster i am trying to run this on, memory and processing power are not limiting. 

Any advice on what might be going wrong here?

Thanks!

This issue appears due to a memory overflow on the number of edges in my graph. Fixing will probably need to wait till igraph 0.10, which should handle 64-bit integers, and so be able to handle far larger graphs 
https://github.com/igraph/python-igraph/issues/531#issuecomment-1102792526 
When the C core `igraph` version 0.10 is released, including a new release of the Python interface building on this new version, i.e. including 64-bit support, I will also update the `leidenalg` implementation to follow suit. In principle, `leidenalg` is already working with 64-bit integers, but since it builds on `igraph`, it is limited by that.
@ivirshup if it's still faster with the same value of `n_iterations`, then why not make the switch?
Could also look at using https://igraph.org/c/doc/igraph-Layout.html#igraph_layout_umap_compute_weights while we're at it with `python-igraph`

Resolving patch:
diff --git a/scanpy/_utils/__init__.py b/scanpy/_utils/__init__.py
--- a/scanpy/_utils/__init__.py
+++ b/scanpy/_utils/__init__.py
@@ -7,9 +7,11 @@
 
 import importlib.util
 import inspect
+import random
 import sys
 import warnings
 from collections import namedtuple
+from contextlib import contextmanager
 from enum import Enum
 from functools import partial, singledispatch, wraps
 from textwrap import dedent
@@ -20,10 +22,10 @@
 import numpy as np
 from anndata import AnnData
 from anndata import __version__ as anndata_version
-from numpy import random
 from numpy.typing import NDArray
 from packaging import version
 from scipy import sparse
+from sklearn.utils import check_random_state
 
 from .. import logging as logg
 from .._compat import DaskArray
@@ -45,7 +47,38 @@ def __repr__(self) -> str:
 _empty = Empty.token
 
 # e.g. https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html
-AnyRandom = Union[int, random.RandomState, None]  # maybe in the future random.Generator
+# maybe in the future random.Generator
+AnyRandom = Union[int, np.random.RandomState, None]
+
+
+class RNGIgraph:
+    """
+    Random number generator for ipgraph so global seed is not changed.
+    See :func:`igraph.set_random_number_generator` for the requirements.
+    """
+
+    def __init__(self, random_state: int = 0) -> None:
+        self._rng = check_random_state(random_state)
+
+    def __getattr__(self, attr: str):
+        return getattr(self._rng, "normal" if attr == "gauss" else attr)
+
+
+@contextmanager
+def set_igraph_random_state(random_state: int):
+    try:
+        import igraph
+    except ImportError:
+        raise ImportError(
+            "Please install igraph: `conda install -c conda-forge igraph` or `pip3 install igraph`."
+        )
+    rng = RNGIgraph(random_state)
+    try:
+        igraph.set_random_number_generator(rng)
+        yield None
+    finally:
+        igraph.set_random_number_generator(random)
+
 
 EPS = 1e-15
 
@@ -459,10 +492,10 @@ def moving_average(a: np.ndarray, n: int):
     return ret[n - 1 :] / n
 
 
-def get_random_state(seed: AnyRandom) -> random.RandomState:
+def get_random_state(seed: AnyRandom) -> np.random.RandomState:
     if isinstance(seed, np.random.RandomState):
         return seed
-    return random.RandomState(seed)
+    return np.random.RandomState(seed)
 
 
 # --------------------------------------------------------------------------------
diff --git a/scanpy/tools/_leiden.py b/scanpy/tools/_leiden.py
--- a/scanpy/tools/_leiden.py
+++ b/scanpy/tools/_leiden.py
@@ -1,6 +1,8 @@
 from __future__ import annotations
 
-from typing import TYPE_CHECKING
+import importlib
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 import numpy as np
 import pandas as pd
@@ -34,14 +36,15 @@ def leiden(
     random_state: _utils.AnyRandom = 0,
     key_added: str = "leiden",
     adjacency: sparse.spmatrix | None = None,
-    directed: bool = True,
+    directed: bool | None = None,
     use_weights: bool = True,
     n_iterations: int = -1,
     partition_type: type[MutableVertexPartition] | None = None,
     neighbors_key: str | None = None,
     obsp: str | None = None,
     copy: bool = False,
-    **partition_kwargs,
+    flavor: Literal["leidenalg", "ipgraph"] = "leidenalg",
+    **clustering_args,
 ) -> AnnData | None:
     """\
     Cluster cells into subgroups [Traag18]_.
@@ -80,6 +83,7 @@ def leiden(
         How many iterations of the Leiden clustering algorithm to perform.
         Positive values above 2 define the total number of iterations to perform,
         -1 has the algorithm run until it reaches its optimal clustering.
+        2 is faster and the default for underlying packages.
     partition_type
         Type of partition to use.
         Defaults to :class:`~leidenalg.RBConfigurationVertexPartition`.
@@ -96,9 +100,11 @@ def leiden(
         `obsp` and `neighbors_key` at the same time.
     copy
         Whether to copy `adata` or modify it inplace.
-    **partition_kwargs
-        Any further arguments to pass to `~leidenalg.find_partition`
-        (which in turn passes arguments to the `partition_type`).
+    flavor
+        Which package's implementation to use.
+    **clustering_args
+        Any further arguments to pass to :func:`~leidenalg.find_partition` (which in turn passes arguments to the `partition_type`)
+        or :meth:`igraph.Graph.community_leiden` from `igraph`.
 
     Returns
     -------
@@ -112,13 +118,35 @@ def leiden(
         A dict with the values for the parameters `resolution`, `random_state`,
         and `n_iterations`.
     """
-    try:
-        import leidenalg
-    except ImportError:
+    if flavor not in {"igraph", "leidenalg"}:
+        raise ValueError(
+            f"flavor must be either 'igraph' or 'leidenalg', but '{flavor}' was passed"
+        )
+    igraph_spec = importlib.util.find_spec("igraph")
+    if igraph_spec is None:
         raise ImportError(
-            "Please install the leiden algorithm: `conda install -c conda-forge leidenalg` or `pip3 install leidenalg`."
+            "Please install the igraph package: `conda install -c conda-forge igraph` or `pip3 install igraph`."
         )
-    partition_kwargs = dict(partition_kwargs)
+    if flavor == "igraph":
+        if directed:
+            raise ValueError(
+                "Cannot use igraph's leiden implemntation with a directed graph."
+            )
+        if partition_type is not None:
+            raise ValueError(
+                "Do not pass in partition_type argument when using igraph."
+            )
+    else:
+        try:
+            import leidenalg
+
+            msg = 'Use of leidenalg is discouraged and will be deprecated in the future.  Please use `flavor="igraph"` `n_iterations=2` to achieve similar results.  `directed` must also be `False` to work with `igraph`\'s implementation.'
+            warnings.warn(msg, FutureWarning)
+        except ImportError:
+            raise ImportError(
+                "Please install the leiden algorithm: `conda install -c conda-forge leidenalg` or `pip3 install leidenalg`."
+            )
+    clustering_args = dict(clustering_args)
 
     start = logg.info("running Leiden clustering")
     adata = adata.copy() if copy else adata
@@ -133,23 +161,29 @@ def leiden(
             restrict_categories=restrict_categories,
             adjacency=adjacency,
         )
-    # convert it to igraph
-    g = _utils.get_igraph_from_adjacency(adjacency, directed=directed)
-    # flip to the default partition type if not overriden by the user
-    if partition_type is None:
-        partition_type = leidenalg.RBConfigurationVertexPartition
     # Prepare find_partition arguments as a dictionary,
     # appending to whatever the user provided. It needs to be this way
     # as this allows for the accounting of a None resolution
     # (in the case of a partition variant that doesn't take it on input)
-    if use_weights:
-        partition_kwargs["weights"] = np.array(g.es["weight"]).astype(np.float64)
-    partition_kwargs["n_iterations"] = n_iterations
-    partition_kwargs["seed"] = random_state
+    clustering_args["n_iterations"] = n_iterations
     if resolution is not None:
-        partition_kwargs["resolution_parameter"] = resolution
-    # clustering proper
-    part = leidenalg.find_partition(g, partition_type, **partition_kwargs)
+        clustering_args["resolution_parameter"] = resolution
+    if flavor == "leidenalg":
+        directed = True if directed is None else directed
+        g = _utils.get_igraph_from_adjacency(adjacency, directed=directed)
+        if partition_type is None:
+            partition_type = leidenalg.RBConfigurationVertexPartition
+        if use_weights:
+            clustering_args["weights"] = np.array(g.es["weight"]).astype(np.float64)
+        clustering_args["seed"] = random_state
+        part = leidenalg.find_partition(g, partition_type, **clustering_args)
+    else:
+        g = _utils.get_igraph_from_adjacency(adjacency, directed=False)
+        if use_weights:
+            clustering_args["weights"] = "weight"
+        clustering_args.setdefault("objective_function", "modularity")
+        with _utils.set_igraph_random_state(random_state):
+            part = g.community_leiden(**clustering_args)
     # store output into adata.obs
     groups = np.array(part.membership)
     if restrict_to is not None:


Test patch:
diff --git a/scanpy/tests/_images/heatmap_var_as_dict/expected.png b/scanpy/tests/_images/heatmap_var_as_dict/expected.png
Binary files a/scanpy/tests/_images/heatmap_var_as_dict/expected.png and b/scanpy/tests/_images/heatmap_var_as_dict/expected.png differ
diff --git a/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_1/expected.png b/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_1/expected.png
Binary files a/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_1/expected.png and b/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_1/expected.png differ
diff --git a/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_2/expected.png b/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_2/expected.png
Binary files a/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_2/expected.png and b/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_2/expected.png differ
diff --git a/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_3/expected.png b/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_3/expected.png
Binary files a/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_3/expected.png and b/scanpy/tests/notebooks/_images_pbmc3k/rank_genes_groups_3/expected.png differ
diff --git a/scanpy/tests/notebooks/_images_pbmc3k/scatter_3/expected.png b/scanpy/tests/notebooks/_images_pbmc3k/scatter_3/expected.png
Binary files a/scanpy/tests/notebooks/_images_pbmc3k/scatter_3/expected.png and b/scanpy/tests/notebooks/_images_pbmc3k/scatter_3/expected.png differ
diff --git a/scanpy/tests/notebooks/_images_pbmc3k/violin_2/expected.png b/scanpy/tests/notebooks/_images_pbmc3k/violin_2/expected.png
Binary files a/scanpy/tests/notebooks/_images_pbmc3k/violin_2/expected.png and b/scanpy/tests/notebooks/_images_pbmc3k/violin_2/expected.png differ
diff --git a/scanpy/tests/notebooks/test_pbmc3k.py b/scanpy/tests/notebooks/test_pbmc3k.py
--- a/scanpy/tests/notebooks/test_pbmc3k.py
+++ b/scanpy/tests/notebooks/test_pbmc3k.py
@@ -28,11 +28,10 @@
 
 @needs.leidenalg
 def test_pbmc3k(image_comparer):
+    # ensure violin plots and other non-determinstic plots have deterministic behavior
+    np.random.seed(0)
     save_and_compare_images = partial(image_comparer, ROOT, tol=20)
-
-    adata = sc.read(
-        "./data/pbmc3k_raw.h5ad", backup_url="https://falexwolf.de/data/pbmc3k_raw.h5ad"
-    )
+    adata = sc.datasets.pbmc3k()
 
     # Preprocessing
 
@@ -105,13 +104,48 @@ def test_pbmc3k(image_comparer):
 
     # Clustering the graph
 
-    sc.tl.leiden(adata, resolution=0.9)
-    # sc.pl.umap(adata, color=['leiden', 'CST3', 'NKG7'], show=False)
-    # save_and_compare_images('umap_2')
+    sc.tl.leiden(
+        adata,
+        resolution=0.9,
+        random_state=0,
+        directed=False,
+        n_iterations=2,
+        flavor="igraph",
+    )
+
+    # sc.pl.umap(adata, color=["leiden", "CST3", "NKG7"], show=False)
+    # save_and_compare_images("umap_2")
     sc.pl.scatter(adata, "CST3", "NKG7", color="leiden", show=False)
     save_and_compare_images("scatter_3")
 
     # Finding marker genes
+    # Due to incosistency with our test runner vs local, these clusters need to
+    # be pre-annotated as the numbers for each cluster are not consistent.
+    marker_genes = [
+        "RP11-18H21.1",
+        "GZMK",
+        "CD79A",
+        "FCGR3A",
+        "GNLY",
+        "S100A8",
+        "FCER1A",
+        "PPBP",
+    ]
+    new_labels = ["0", "1", "2", "3", "4", "5", "6", "7"]
+    data_df = adata[:, marker_genes].to_df()
+    data_df["leiden"] = adata.obs["leiden"]
+    max_idxs = data_df.groupby("leiden", observed=True).mean().idxmax()
+    leiden_relabel = {}
+    for marker_gene, new_label in zip(marker_genes, new_labels):
+        leiden_relabel[max_idxs[marker_gene]] = new_label
+    adata.obs["leiden_old"] = adata.obs["leiden"].copy()
+    adata.rename_categories(
+        "leiden", [leiden_relabel[key] for key in sorted(leiden_relabel.keys())]
+    )
+    # ensure that the column can be sorted for consistent plotting since it is by default unordered
+    adata.obs["leiden"] = adata.obs["leiden"].cat.reorder_categories(
+        list(map(str, range(len(adata.obs["leiden"].cat.categories)))), ordered=True
+    )
 
     sc.tl.rank_genes_groups(adata, "leiden")
     sc.pl.rank_genes_groups(adata, n_genes=20, sharey=False, show=False)
@@ -129,18 +163,13 @@ def test_pbmc3k(image_comparer):
     # sc.pl.rank_genes_groups_violin(adata, groups='0', n_genes=8)
     # save_and_compare_images('rank_genes_groups_4')
 
-    if adata[adata.obs["leiden"] == "4", "CST3"].X.mean() < 1:
-        (  # switch clusters
-            adata.obs["leiden"][adata.obs["leiden"] == "4"],
-            adata.obs["leiden"][adata.obs["leiden"] == "5"],
-        ) = ("5", "4")
     new_cluster_names = [
         "CD4 T cells",
-        "CD14+ Monocytes",
-        "B cells",
         "CD8 T cells",
+        "B cells",
         "NK cells",
         "FCGR3A+ Monocytes",
+        "CD14+ Monocytes",
         "Dendritic cells",
         "Megakaryocytes",
     ]
@@ -148,7 +177,6 @@ def test_pbmc3k(image_comparer):
 
     # sc.pl.umap(adata, color='leiden', legend_loc='on data', title='', frameon=False, show=False)
     # save_and_compare_images('umap_3')
-
     sc.pl.violin(
         adata, ["CST3", "NKG7", "PPBP"], groupby="leiden", rotation=90, show=False
     )
diff --git a/scanpy/tests/test_clustering.py b/scanpy/tests/test_clustering.py
--- a/scanpy/tests/test_clustering.py
+++ b/scanpy/tests/test_clustering.py
@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import pytest
+from sklearn.metrics.cluster import normalized_mutual_info_score
 
 import scanpy as sc
 from scanpy.testing._helpers.data import pbmc68k_reduced
@@ -12,11 +13,136 @@ def adata_neighbors():
     return pbmc68k_reduced()
 
 
+FLAVORS = [
+    pytest.param("igraph", marks=needs.igraph),
+    pytest.param("leidenalg", marks=needs.leidenalg),
+]
+
+
+@needs.leidenalg
+@needs.igraph
+@pytest.mark.parametrize("flavor", FLAVORS)
+@pytest.mark.parametrize("resolution", [1, 2])
+@pytest.mark.parametrize("n_iterations", [-1, 3])
+def test_leiden_basic(adata_neighbors, flavor, resolution, n_iterations):
+    sc.tl.leiden(
+        adata_neighbors,
+        flavor=flavor,
+        resolution=resolution,
+        n_iterations=n_iterations,
+        directed=(flavor == "leidenalg"),
+    )
+    assert adata_neighbors.uns["leiden"]["params"]["resolution"] == resolution
+    assert adata_neighbors.uns["leiden"]["params"]["n_iterations"] == n_iterations
+
+
 @needs.leidenalg
-def test_leiden_basic(adata_neighbors):
-    sc.tl.leiden(adata_neighbors)
+@needs.igraph
+@pytest.mark.parametrize("flavor", FLAVORS)
+def test_leiden_random_state(adata_neighbors, flavor):
+    is_leiden_alg = flavor == "leidenalg"
+    n_iterations = 2 if is_leiden_alg else -1
+    adata_1 = sc.tl.leiden(
+        adata_neighbors,
+        flavor=flavor,
+        random_state=1,
+        copy=True,
+        directed=is_leiden_alg,
+        n_iterations=n_iterations,
+    )
+    adata_1_again = sc.tl.leiden(
+        adata_neighbors,
+        flavor=flavor,
+        random_state=1,
+        copy=True,
+        directed=is_leiden_alg,
+        n_iterations=n_iterations,
+    )
+    adata_2 = sc.tl.leiden(
+        adata_neighbors,
+        flavor=flavor,
+        random_state=2,
+        copy=True,
+        directed=is_leiden_alg,
+        n_iterations=n_iterations,
+    )
+    assert (adata_1.obs["leiden"] == adata_1_again.obs["leiden"]).all()
+    assert (adata_2.obs["leiden"] != adata_1_again.obs["leiden"]).any()
+
+
+@needs.igraph
+def test_leiden_igraph_directed(adata_neighbors):
+    with pytest.raises(ValueError):
+        sc.tl.leiden(adata_neighbors, flavor="igraph", directed=True)
+
+
+@needs.igraph
+def test_leiden_wrong_flavor(adata_neighbors):
+    with pytest.raises(ValueError):
+        sc.tl.leiden(adata_neighbors, flavor="foo")
+
+
+@needs.igraph
+@needs.leidenalg
+def test_leiden_igraph_partition_type(adata_neighbors):
+    import leidenalg
+
+    with pytest.raises(ValueError):
+        sc.tl.leiden(
+            adata_neighbors,
+            flavor="igraph",
+            partition_type=leidenalg.RBConfigurationVertexPartition,
+        )
+
+
+@needs.leidenalg
+@needs.igraph
+def test_leiden_equal_defaults_same_args(adata_neighbors):
+    """Ensure the two implementations are the same for the same args."""
+    leiden_alg_clustered = sc.tl.leiden(
+        adata_neighbors, flavor="leidenalg", copy=True, n_iterations=2
+    )
+    igraph_clustered = sc.tl.leiden(
+        adata_neighbors, flavor="igraph", copy=True, directed=False, n_iterations=2
+    )
+    assert (
+        normalized_mutual_info_score(
+            leiden_alg_clustered.obs["leiden"], igraph_clustered.obs["leiden"]
+        )
+        > 0.9
+    )
+
+
+@needs.leidenalg
+@needs.igraph
+def test_leiden_equal_defaults(adata_neighbors):
+    """Ensure that the old leidenalg defaults are close enough to the current default outputs."""
+    leiden_alg_clustered = sc.tl.leiden(
+        adata_neighbors, flavor="leidenalg", directed=True, copy=True
+    )
+    igraph_clustered = sc.tl.leiden(
+        adata_neighbors, copy=True, n_iterations=2, directed=False
+    )
+    assert (
+        normalized_mutual_info_score(
+            leiden_alg_clustered.obs["leiden"], igraph_clustered.obs["leiden"]
+        )
+        > 0.9
+    )
+
+
+@needs.igraph
+def test_leiden_objective_function(adata_neighbors):
+    """Ensure that popping this as a `clustering_kwargs` and using it does not error out."""
+    sc.tl.leiden(
+        adata_neighbors,
+        objective_function="modularity",
+        flavor="igraph",
+        directed=False,
+    )
 
 
+@needs.igraph
 @pytest.mark.parametrize(
     "clustering,key",
     [
@@ -52,6 +178,7 @@ def test_clustering_subset(adata_neighbors, clustering, key):
 
 
 @needs.louvain
+@needs.igraph
 def test_louvain_basic(adata_neighbors):
     sc.tl.louvain(adata_neighbors)
     sc.tl.louvain(adata_neighbors, use_weights=True)
@@ -60,6 +187,7 @@ def test_louvain_basic(adata_neighbors):
 
 
 @needs.louvain
+@needs.igraph
 def test_partition_type(adata_neighbors):
     import louvain
 
diff --git a/scanpy/tests/test_plotting.py b/scanpy/tests/test_plotting.py
--- a/scanpy/tests/test_plotting.py
+++ b/scanpy/tests/test_plotting.py
@@ -103,7 +103,14 @@ def test_heatmap(image_comparer):
 
     # test var_names as dict
     pbmc = pbmc68k_reduced()
-    sc.tl.leiden(pbmc, key_added="clusters", resolution=0.5)
+    sc.tl.leiden(
+        pbmc,
+        key_added="clusters",
+        resolution=0.5,
+        flavor="igraph",
+        n_iterations=2,
+        directed=False,
+    )
     # call umap to trigger colors for the clusters
     sc.pl.umap(pbmc, color="clusters")
     marker_genes_dict = {