Repository: networkx/networkx
Pull request #7949: Fix BC scaling for source nodes with k and endpoints=False
Author: login: eriknw
Labels: name: type: Bug fix
Created: 2025-03-31T15:22:52Z | Merged: 2025-04-01T02:41:36Z

This is a follow up to #7908 and specifically @dschult's question from https://github.com/networkx/networkx/pull/7908#pullrequestreview-2711295366:
> The case with k < n and endpoints is False still confuses me. It seems like the number of valid (s, t) node pairs that could have a path through v where s != t should depend on whether node v was in the sample or not in the sample. The largest possible count for nodes in the sample (when endpoints is False) is (k-1)*(n-2), but for nodes not in the sample the count can be k*(n-2). Why are we using k*(n-2) for all the nodes?

After investigating this, I think source nodes and non-source nodes do indeed need to scale differently when `k` is given and `endpoints=False`.

I added a test that exercises this. I don't fully grok the test results, but I trust the scaling logic more than I did.

CC @ChuckHastings @rlratzel 


Diff:
diff --git a/networkx/algorithms/centrality/betweenness.py b/networkx/algorithms/centrality/betweenness.py
index df8f21d10ee..eca773c064b 100644
--- a/networkx/algorithms/centrality/betweenness.py
+++ b/networkx/algorithms/centrality/betweenness.py
@@ -1,5 +1,6 @@
 """Betweenness centrality measures."""
 
+import math
 from collections import deque
 from heapq import heappop, heappush
 from itertools import count
@@ -128,10 +129,7 @@ def betweenness_centrality(
     """
     betweenness = dict.fromkeys(G, 0.0)  # b[v]=0 for v in G
     if k == len(G):
-        # This is for performance and correctness. When `endpoints` is False
-        # and k is given, k == n is a special case that would violate the
-        # assumption that node `v` is not one of the (s, t) node pairs.
-        # Should this warn or raise instead?
+        # This is done for performance; the result is the same regardless.
         k = None
     if k is None:
         nodes = G
@@ -156,6 +154,7 @@ def betweenness_centrality(
         directed=G.is_directed(),
         k=k,
         endpoints=endpoints,
+        sampled_nodes=nodes,
     )
     return betweenness
 
@@ -362,7 +361,7 @@ def _accumulate_edges(betweenness, S, P, sigma, s):
     return betweenness
 
 
-def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False):
+def _rescale(betweenness, n, *, normalized, directed, k, endpoints, sampled_nodes):
     # N is used to count the number of valid (s, t) pairs where s != t that
     # could have a path pass through v. If endpoints is False, then v must
     # not be the target t, hence why we subtract by 1.
@@ -372,24 +371,48 @@ def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False
         return betweenness
 
     K_source = N if k is None else k
+
+    if k is None or endpoints:
+        # No sampling adjustment needed
+        if normalized:
+            # Divide by the number of valid (s, t) node pairs that could have
+            # a path through v where s != t.
+            scale = 1 / (K_source * (N - 1))
+        else:
+            # Scale to the full BC
+            if not directed:
+                # The non-normalized BC values are computed the same way for
+                # directed and undirected graphs: shortest paths are computed and
+                # counted for each *ordered* (s, t) pair. Undirected graphs should
+                # only count valid *unordered* node pairs {s, t}; that is, (s, t)
+                # and (t, s) should be counted only once. We correct for this here.
+                correction = 2
+            else:
+                correction = 1
+            scale = N / (K_source * correction)
+
+        if scale != 1:
+            for v in betweenness:
+                betweenness[v] *= scale
+        return betweenness
+
+    # Sampling adjustment needed when excluding endpoints when using k. In this
+    # case, we need to handle source nodes differently from non-source nodes,
+    # because source nodes can't include themselves since endpoints are excluded.
+    # Without this, k == n would be a special case that would violate the
+    # assumption that node `v` is not one of the (s, t) node pairs.
     if normalized:
-        # Divide by the number of valid (s, t) node pairs that could have
-        # a path through v where s != t.
-        scale = 1 / (K_source * (N - 1))
+        # NaN for undefined 0/0; there is no data for source node when k=1
+        scale_source = 1 / ((K_source - 1) * (N - 1)) if K_source > 1 else math.nan
+        scale_nonsource = 1 / (K_source * (N - 1))
     else:
-        if not directed:
-            # The non-normalized BC values are computed the same way for
-            # directed and undirected graphs: shortest paths are computed and
-            # counted for each *ordered* (s, t) pair. Undirected graphs should
-            # only count valid *unordered* node pairs {s, t}; that is, (s, t)
-            # and (t, s) should be counted only once. We correct for this here.
-            K_source *= 2
-        # Scale to the full BC
-        scale = N / K_source
-
-    if scale != 1:
-        for v in betweenness:
-            betweenness[v] *= scale
+        correction = 1 if directed else 2
+        scale_source = N / ((K_source - 1) * correction) if K_source > 1 else math.nan
+        scale_nonsource = N / (K_source * correction)
+
+    sampled_nodes = set(sampled_nodes)
+    for v in betweenness:
+        betweenness[v] *= scale_source if v in sampled_nodes else scale_nonsource
     return betweenness
 
 
diff --git a/networkx/algorithms/centrality/tests/test_betweenness_centrality.py b/networkx/algorithms/centrality/tests/test_betweenness_centrality.py
index d9233811268..5c988ae67b1 100644
--- a/networkx/algorithms/centrality/tests/test_betweenness_centrality.py
+++ b/networkx/algorithms/centrality/tests/test_betweenness_centrality.py
@@ -1,3 +1,5 @@
+import math
+
 import pytest
 
 import networkx as nx
@@ -295,18 +297,18 @@ def test_directed_path_normalized(self):
             (True, True, True, 1, {0: 1.0, 1: 1.0, 2: 0.25, 3: 0.25, 4: 0.25}),
             (True, True, False, None, {0: 1.0, 1: 0.4, 2: 0.4, 3: 0.4, 4: 0.4}),
             (True, True, False, 1, {0: 1.0, 1: 1.0, 2: 0.25, 3: 0.25, 4: 0.25}),
-            (True, False, True, None, {0: 1.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}),
-            (True, False, True, 1, {0: 1.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}),
+            (True, False, True, None, {0: 1.0, 1: 0, 2: 0.0, 3: 0.0, 4: 0.0}),
+            (True, False, True, 1, {0: 1.0, 1: math.nan, 2: 0.0, 3: 0.0, 4: 0.0}),
             (True, False, False, None, {0: 1.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}),
-            (True, False, False, 1, {0: 1.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}),
+            (True, False, False, 1, {0: 1.0, 1: math.nan, 2: 0.0, 3: 0.0, 4: 0.0}),
             (False, True, True, None, {0: 20.0, 1: 8.0, 2: 8.0, 3: 8.0, 4: 8.0}),
             (False, True, True, 1, {0: 20.0, 1: 20.0, 2: 5.0, 3: 5.0, 4: 5.0}),
             (False, True, False, None, {0: 10.0, 1: 4.0, 2: 4.0, 3: 4.0, 4: 4.0}),
             (False, True, False, 1, {0: 10.0, 1: 10.0, 2: 2.5, 3: 2.5, 4: 2.5}),
             (False, False, True, None, {0: 12.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}),
-            (False, False, True, 1, {0: 12.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}),
+            (False, False, True, 1, {0: 12.0, 1: math.nan, 2: 0.0, 3: 0.0, 4: 0.0}),
             (False, False, False, None, {0: 6.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}),
-            (False, False, False, 1, {0: 6.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}),
+            (False, False, False, 1, {0: 6.0, 1: math.nan, 2: 0.0, 3: 0.0, 4: 0.0}),
         ],
     )
     def test_scale_with_k_on_star_graph(
@@ -320,6 +322,77 @@ def test_scale_with_k_on_star_graph(
         b = nx.betweenness_centrality(
             G, k=k, seed=1, endpoints=endpoints, normalized=normalized
         )
+        assert b == pytest.approx(expected, nan_ok=True)
+
+    @pytest.mark.parametrize(
+        ("normalized", "endpoints", "is_directed", "k", "expected"),
+        [
+            (
+                *(True, True, True, None),  # Use *() splatting for better autoformat
+                {0: 14 / 20, 1: 14 / 20, 2: 14 / 20, 3: 14 / 20, 4: 14 / 20},
+            ),
+            (
+                *(True, True, True, 3),
+                {0: 9 / 12, 1: 11 / 12, 2: 9 / 12, 3: 6 / 12, 4: 7 / 12},
+            ),
+            (
+                *(True, True, False, None),
+                {0: 10 / 20, 1: 10 / 20, 2: 10 / 20, 3: 10 / 20, 4: 10 / 20},
+            ),
+            (
+                *(True, True, False, 3),
+                {0: 8 / 12, 1: 7 / 12, 2: 4 / 12, 3: 4 / 12, 4: 7 / 12},
+            ),
+            (
+                *(True, False, True, None),
+                {0: 6 / 12, 1: 6 / 12, 2: 6 / 12, 3: 6 / 12, 4: 6 / 12},
+            ),
+            (
+                *(True, False, True, 3),
+                # Use 6 instead of 9 for denominator for source nodes 0, 1, and 4
+                {0: 3 / 6, 1: 5 / 6, 2: 6 / 9, 3: 3 / 9, 4: 1 / 6},
+            ),
+            (
+                *(True, False, False, None),
+                {0: 2 / 12, 1: 2 / 12, 2: 2 / 12, 3: 2 / 12, 4: 2 / 12},
+            ),
+            (
+                *(True, False, False, 3),
+                # Use 6 instead of 9 for denominator for source nodes 0, 1, and 4
+                {0: 2 / 6, 1: 1 / 6, 2: 1 / 9, 3: 1 / 9, 4: 1 / 6},
+            ),
+            (False, True, True, None, {0: 14, 1: 14, 2: 14, 3: 14, 4: 14}),
+            (
+                *(False, True, True, 3),
+                {0: 9 * 5 / 3, 1: 11 * 5 / 3, 2: 9 * 5 / 3, 3: 6 * 5 / 3, 4: 7 * 5 / 3},
+            ),
+            (False, True, False, None, {0: 5, 1: 5, 2: 5, 3: 5, 4: 5}),
+            (
+                *(False, True, False, 3),
+                {0: 8 * 5 / 6, 1: 7 * 5 / 6, 2: 4 * 5 / 6, 3: 4 * 5 / 6, 4: 7 * 5 / 6},
+            ),
+            (False, False, True, None, {0: 6, 1: 6, 2: 6, 3: 6, 4: 6}),
+            (
+                *(False, False, True, 3),
+                # Use 2 instead of 3 for denominator for source nodes 0, 1, and 4
+                {0: 3 * 4 / 2, 1: 5 * 4 / 2, 2: 6 * 4 / 3, 3: 3 * 4 / 3, 4: 1 * 4 / 2},
+            ),
+            (False, False, False, None, {0: 1, 1: 1, 2: 1, 3: 1, 4: 1}),
+            (
+                *(False, False, False, 3),
+                # Use 4 instead of 6 for denominator for source nodes 0, 1, and 4
+                {0: 2 * 4 / 4, 1: 1 * 4 / 4, 2: 1 * 4 / 6, 3: 1 * 4 / 6, 4: 1 * 4 / 4},
+            ),
+        ],
+    )
+    def test_scale_with_k_on_cycle_graph(
+        self, normalized, endpoints, is_directed, k, expected
+    ):
+        # seed=1 selects nodes 0, 1, and 4 as the initial nodes when using k=3.
+        G = nx.cycle_graph(5, create_using=nx.DiGraph if is_directed else nx.Graph)
+        b = nx.betweenness_centrality(
+            G, k=k, seed=1, endpoints=endpoints, normalized=normalized
+        )
         assert b == pytest.approx(expected)
 
     def test_k_out_of_bounds_raises(self):


Reviews:
user: login: eriknw
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T15:25:45Z
user: login: amcandio
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T15:39:00Z
user: login: eriknw
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T15:49:14Z
user: login: dschult
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T15:49:58Z
user: login: amcandio
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T17:56:53Z
user: login: eriknw
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T19:19:27Z
user: login: dschult
body: Thanks a lot for this @eriknw!!
Especially for the tests and making them readable! :)

I have some optional tweaks and suggestions below that I hope keep the flow clear.

I think the main issues I have left are:
- when `k=1 and endpoints=False` should we raise? or assign to the sampled node `0` or `'nan'`? See comment below.
- how to handle undirected paths when sampling. I will try to explain why I think we are using the right formulas even though the appear to ignore double-counting in the sample. The key is assuming that the starting nodes are chosen randomly so chosen paths are not related to double-counting.

**Undirected paths with `k<n`**:
For undirected paths, I have convinced myself that what we are doing is the best we can do unless we start tracking which paths are double-counted. And I don't see an easy way to track that with the Brandes sigma/delta formulation.  Here's my justification:

Let `n=len(G)` and `N = n if endpoints else n-1` and `K = k-1 if node in sample else k`. Let`bc_raw` be the count of paths through a node, and `bcf` be the actual betweenness value normalized so it is the fraction of al possible paths that pass through a node. Let `bc_count` be the unnormalized value. I'll use "distinct paths" to refer to undirected paths that don't double-count both directions.

We currently use `bc_count = bc_raw * N/(2*K)` for undirected graphs. This works well for full `bc` computations, but it seems wrong for sampling. 

The number of possible distinct paths through a node is `N*(N-1)/2`, but the number of distinct paths for a sample of `k` starting nodes is not as simple. Our sample has `K*(K-1)/2` possible double-counted paths for each node. So the number of distinct paths in our sample is `K(N-1) - K(K-1)/2`. How then can we approximate the count of distinct paths through a node?

Let's assume that a path eventually double-counted is just as likely pass through this node as a single-counted path (which should be true for random sampling of starting nodes). 

Then the expected number of double-counted paths should be `bcf*K*(K-1)/2`. Here bcf is the true value -- not an estimate. Our estimate of it comes from `bc_raw` in the relation `bcf = (bc_raw - bcf*K*(K-1)/2) / (K*(N-1) - K*(K-1)/2)`. That is the ratio of counted distinct paths to possible distinct paths. This implicit equation for `bcf` can be rearranged to solve for `bcf` as 

```bcf = bc_raw/(K*(N-1))```

But that is the same formula as for directed graphs!! So just as we currently do, we should compute the normalized bc values without regard to double-counting paths.

To get the non-normalized `bc_count` from `bcf`, multiply by the total possible distinct paths: 
```bc_count = bcf * N*(N-1)/2 = bc_raw * N/(2*K)`.
This again is our current formula. The factor of 2 is the same.

I believe that to do better than this we need to track which paths through each node are double counted. And I think that would disturb our Brandes-based speed-up in counting the paths.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T20:37:31Z
user: login: dschult
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T20:45:30Z
user: login: dschult
body: Niiccee!!!

Thanks @eriknw

I approve this PR
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-31T22:15:38Z
user: login: rossbar
body: Thanks all, again, for the detailed dive into the guts of betweenness centrality. I still can't claim to 100% follow all of the details, but the introduced tests really help me understand what's going on!

Once this work is completed and all follow-ups are in, I'm definitely in favor of a detailed release note to explain what all changed and why!

Finally - I have one nit comment: should we use `math.nan` or `float("nan")`? I don't really know whether there's a difference and am happy with whatever we decide, but figured I'd raise it for my own edification!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T22:38:00Z
user: login: eriknw
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T00:15:55Z
user: login: eriknw
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T00:16:06Z
user: login: eriknw
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T00:16:27Z
user: login: eriknw
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T00:17:35Z

Review comments:
diff_hunk: @@ -128,10 +128,7 @@ def betweenness_centrality(
     """
     betweenness = dict.fromkeys(G, 0.0)  # b[v]=0 for v in G
     if k == len(G):
-        # This is for performance and correctness. When `endpoints` is False
-        # and k is given, k == n is a special case that would violate the
-        # assumption that node `v` is not one of the (s, t) node pairs.
-        # Should this warn or raise instead?
+        # This is done for performance; the result is the same regardless.
path: networkx/algorithms/centrality/betweenness.py
user: login: eriknw
body: It's nice that `k == n` is no longer a "special case", which otherwise is a red flag. Now, if we kept `k = n`, we would get the same result! This makes sense to me. This should be comforting and highly suggestive that scaling should be done as in this PR.
created_at: 2025-03-31T15:25:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -371,23 +371,50 @@ def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False
         # No rescaling necessary: b=0 for all nodes
         return betweenness
 
+    adjust_initial_samples = k is not None and nodes and not endpoints
+    if adjust_initial_samples:
+        nodes = set(nodes)
+
     K_source = N if k is None else k
     if normalized:
         # Divide by the number of valid (s, t) node pairs that could have
         # a path through v where s != t.
-        scale = 1 / (K_source * (N - 1))
+        if not adjust_initial_samples:
+            scale = 1 / (K_source * (N - 1))
+        else:
+            # Branch for excluding endpoints when using k. In this case, we need
+            # to handle source nodes differently from non-source nodes, because
+            # source nodes can't include themselves since endpoints are excluded.
+            # Without this, k == n would be a special case that would violate
+            # the assumption that node `v` is not one of the (s, t) node pairs.
+            # Avoid ZeroDivisionError when K_source == 1.
+            scale_source = 1 / ((K_source - 1) * (N - 1)) if K_source > 1 else 1
+            scale_nonsource = 1 / (K_source * (N - 1))
path: networkx/algorithms/centrality/betweenness.py
user: login: amcandio
body: Can N be 1? checking for div 0 errors
created_at: 2025-03-31T15:38:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -371,23 +371,50 @@ def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False
         # No rescaling necessary: b=0 for all nodes
         return betweenness
 
+    adjust_initial_samples = k is not None and nodes and not endpoints
+    if adjust_initial_samples:
+        nodes = set(nodes)
+
     K_source = N if k is None else k
     if normalized:
         # Divide by the number of valid (s, t) node pairs that could have
         # a path through v where s != t.
-        scale = 1 / (K_source * (N - 1))
+        if not adjust_initial_samples:
+            scale = 1 / (K_source * (N - 1))
+        else:
+            # Branch for excluding endpoints when using k. In this case, we need
+            # to handle source nodes differently from non-source nodes, because
+            # source nodes can't include themselves since endpoints are excluded.
+            # Without this, k == n would be a special case that would violate
+            # the assumption that node `v` is not one of the (s, t) node pairs.
+            # Avoid ZeroDivisionError when K_source == 1.
+            scale_source = 1 / ((K_source - 1) * (N - 1)) if K_source > 1 else 1
+            scale_nonsource = 1 / (K_source * (N - 1))
path: networkx/algorithms/centrality/betweenness.py
user: login: eriknw
body: Good question, thanks for reviewing! `N` is at least 2 once this code is reached. We handle `N < 2` at the beginning of the function and return w/o scaling, because the BC values will all be 0.
created_at: 2025-03-31T15:49:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -128,10 +128,7 @@ def betweenness_centrality(
     """
     betweenness = dict.fromkeys(G, 0.0)  # b[v]=0 for v in G
     if k == len(G):
-        # This is for performance and correctness. When `endpoints` is False
-        # and k is given, k == n is a special case that would violate the
-        # assumption that node `v` is not one of the (s, t) node pairs.
-        # Should this warn or raise instead?
+        # This is done for performance; the result is the same regardless.
path: networkx/algorithms/centrality/betweenness.py
user: login: dschult
body: Yes!  I agree...   And I was very much looking forward to having this comment about "correctness" removed. It does indeed make me feel more comfortable.

There is another part of this that I was wrestling with over the weekend. Instead of adjusting the scaling factor based on whether a node is in the sample or not, could we replace `k` with the expected value of the number of paths?  That is, each node has chance `k/n` of being in the sample, so instead of `k` and `k-1` for the different nodes, we could use `k - k/n`.  That then is `k*(n-1)/n` which turns us around to scaling by `n/k` instead of `(n-1)/k`.  This feels like it reverts some of the changes we made in the previous PR. But -- because we do have the set of nodes that are sampled, I prefer using that information for scaling. I guess I don't trust the "Law of large numbers " when we can use the more precise small numbers information.
 
I was working on this, but got distracted by our `undirected graph` adjustment when sampling.
I think there are troubles there too.  Perhaps this is better for yet another PR though.
When `k=1` we know that we have not double-counted any undirected paths. So we should NOT be dividing the count by 2. When `k=2` we know that we considered at most 1 double-counted path. In general, we consider `k(k-1)/2` paths twice, and the remaining `k*(n-2) - k(k-1)/2` paths once. There must be a way to adjust our scaling of the undirected case to account for this bias when sampling. 
created_at: 2025-03-31T15:49:58Z
author_association: MEMBER
diff_hunk: @@ -371,23 +371,50 @@ def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False
         # No rescaling necessary: b=0 for all nodes
         return betweenness
 
+    adjust_initial_samples = k is not None and nodes and not endpoints
+    if adjust_initial_samples:
+        nodes = set(nodes)
+
     K_source = N if k is None else k
     if normalized:
         # Divide by the number of valid (s, t) node pairs that could have
         # a path through v where s != t.
-        scale = 1 / (K_source * (N - 1))
+        if not adjust_initial_samples:
+            scale = 1 / (K_source * (N - 1))
+        else:
+            # Branch for excluding endpoints when using k. In this case, we need
+            # to handle source nodes differently from non-source nodes, because
+            # source nodes can't include themselves since endpoints are excluded.
+            # Without this, k == n would be a special case that would violate
+            # the assumption that node `v` is not one of the (s, t) node pairs.
+            # Avoid ZeroDivisionError when K_source == 1.
+            scale_source = 1 / ((K_source - 1) * (N - 1)) if K_source > 1 else 1
+            scale_nonsource = 1 / (K_source * (N - 1))
path: networkx/algorithms/centrality/betweenness.py
user: login: amcandio
body: Got it thanks! LGTM
created_at: 2025-03-31T17:56:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -371,23 +371,50 @@ def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False
         # No rescaling necessary: b=0 for all nodes
         return betweenness
 
+    adjust_initial_samples = k is not None and nodes and not endpoints
+    if adjust_initial_samples:
+        nodes = set(nodes)
+
     K_source = N if k is None else k
     if normalized:
         # Divide by the number of valid (s, t) node pairs that could have
         # a path through v where s != t.
-        scale = 1 / (K_source * (N - 1))
+        if not adjust_initial_samples:
+            scale = 1 / (K_source * (N - 1))
+        else:
+            # Branch for excluding endpoints when using k. In this case, we need
+            # to handle source nodes differently from non-source nodes, because
+            # source nodes can't include themselves since endpoints are excluded.
+            # Without this, k == n would be a special case that would violate
+            # the assumption that node `v` is not one of the (s, t) node pairs.
+            # Avoid ZeroDivisionError when K_source == 1.
+            scale_source = 1 / ((K_source - 1) * (N - 1)) if K_source > 1 else 1
path: networkx/algorithms/centrality/betweenness.py
user: login: dschult
body: How should we handle the case when k=1 and the node was in the sample. The number of counted paths through the node is 0, and the number of possible paths passing through the node is 0.  So we get `0/0`.  That makes sense because we actually have no sample to make an estimate for this node!  Without any data, what should we estimate?

We could:
- raise when k=1, end_points=False
- return `nan` for the node chosen as the starting node
- return 0 for the node chosen as the starting node (this is what this PR does I think)

I have a slight preference for changing the `1` at the end of this line to `nan`. But I could be persuaded... especially for raising when k=1 and endpoints=False. It feels funny to return `0` when we don't know anything about the paths through this node.
created_at: 2025-03-31T19:19:17Z
author_association: MEMBER
diff_hunk: @@ -128,10 +128,7 @@ def betweenness_centrality(
     """
     betweenness = dict.fromkeys(G, 0.0)  # b[v]=0 for v in G
     if k == len(G):
-        # This is for performance and correctness. When `endpoints` is False
-        # and k is given, k == n is a special case that would violate the
-        # assumption that node `v` is not one of the (s, t) node pairs.
-        # Should this warn or raise instead?
+        # This is done for performance; the result is the same regardless.
path: networkx/algorithms/centrality/betweenness.py
user: login: eriknw
body: I think I understand your two points.

For your middle paragraph, I think you are suggesting an alternative that uses an expectation or an average value to use for the scale for all the nodes, but still prefer the exactness (and probably correctness) for scaling each node depending whether it was a source or not. Yeah, if we didn't have the source nodes available, I think this would be better than nothing, but we might as well use the source node information since we have it.

I'm not sure I completely follow your last paragraph, so let me paraphrase and correct me if I'm wrong. Basically, for undirected graphs when using k without normalizing, you are considering changing the "double counting" correction. The intuition is that for k=1, then no `(s, t)` and `(t, s)` pairs are double-counted, so we don't need to divide by two, and for k=len(G), then all `(s, t)` and `(t, s)` pairs were double-counted exactly, so we need to divide by two for all nodes. A linear scaling between these two boundaries could be `correction = 1 + (k - 1) / (n - 1)` (I almost certainly got this wrong), but maybe the more accurate way to count this would be to divide contributions of paths by two if `(s, t)` and `(t, s)` pairs were both sampled, but don't divide by two for paths if there is not a `(s, t)` and `(t, s)` pair. I don't think I buy this; or, rather, I don't think my current understanding of it is correct. I'm not able to articulate this yet, so I'm not confident, but I think we already do a good job of accounting for possible paths. I would be curious if there's an algorithm or implementation for undirected graphs with k samples that does not require the double-counting correction.
created_at: 2025-03-31T19:19:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -362,7 +360,9 @@ def _accumulate_edges(betweenness, S, P, sigma, s):
     return betweenness
 
 
-def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False):
+def _rescale(
+    betweenness, n, normalized, directed=False, k=None, endpoints=False, nodes=None
path: networkx/algorithms/centrality/betweenness.py
user: login: dschult
body: Can we rename `nodes` as `sampled_nodes`?
And let's remove the default values. This helper function doesn't need defaults. (makes the signature line shorter too.)
created_at: 2025-03-31T19:20:48Z
author_association: MEMBER
diff_hunk: @@ -371,23 +371,50 @@ def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False
         # No rescaling necessary: b=0 for all nodes
         return betweenness
 
+    adjust_initial_samples = k is not None and nodes and not endpoints
path: networkx/algorithms/centrality/betweenness.py
user: login: dschult
body: Can `nodes` ever be not true? I guess if `k=0`, but if that's the reason here, let's catch that at the very beginning of the main function.

I prefer the name `adjust_for_sampling`... but I'm hoping we don't need this at all if we return directly after scaling (see comment below).  We'll still need to check this -- and turn `nodes` into a set. But only in two places and it'll make the flow more clear, I hope.
created_at: 2025-03-31T19:30:27Z
author_association: MEMBER
diff_hunk: @@ -371,23 +371,50 @@ def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False
         # No rescaling necessary: b=0 for all nodes
         return betweenness
 
+    adjust_initial_samples = k is not None and nodes and not endpoints
+    if adjust_initial_samples:
+        nodes = set(nodes)
+
     K_source = N if k is None else k
     if normalized:
         # Divide by the number of valid (s, t) node pairs that could have
         # a path through v where s != t.
-        scale = 1 / (K_source * (N - 1))
+        if not adjust_initial_samples:
+            scale = 1 / (K_source * (N - 1))
+        else:
+            # Branch for excluding endpoints when using k. In this case, we need
+            # to handle source nodes differently from non-source nodes, because
+            # source nodes can't include themselves since endpoints are excluded.
+            # Without this, k == n would be a special case that would violate
+            # the assumption that node `v` is not one of the (s, t) node pairs.
+            # Avoid ZeroDivisionError when K_source == 1.
+            scale_source = 1 / ((K_source - 1) * (N - 1)) if K_source > 1 else 1
+            scale_nonsource = 1 / (K_source * (N - 1))
+            for v in betweenness:
+                betweenness[v] *= scale_source if v in nodes else scale_nonsource
     else:
         if not directed:
             # The non-normalized BC values are computed the same way for
             # directed and undirected graphs: shortest paths are computed and
             # counted for each *ordered* (s, t) pair. Undirected graphs should
             # only count valid *unordered* node pairs {s, t}; that is, (s, t)
             # and (t, s) should be counted only once. We correct for this here.
-            K_source *= 2
+            correction = 2
+        else:
+            correction = 1
         # Scale to the full BC
-        scale = N / K_source
-
-    if scale != 1:
+        if not adjust_initial_samples:
+            scale = N / (K_source * correction)
+        else:
+            # Like above, we need to handle source nodes and non-source nodes
+            # differently when using k while excluding endpoints.
+            # Avoid ZeroDivisionError when K_source == 1.
+            scale_source = N / ((K_source - 1) * correction) if K_source > 1 else 1
+            scale_nonsource = N / (K_source * correction)
+            for v in betweenness:
+                betweenness[v] *= scale_source if v in nodes else scale_nonsource
+
+    if not adjust_initial_samples and scale != 1:
path: networkx/algorithms/centrality/betweenness.py
user: login: dschult
body: I'm envisioning this as:
- check `is not None and not endpoints` directly instead of using a flag.
- make `nodes` a set just before we need it to be a set
- change `1` to `float('nan')`
- return immediately (which also makes the final `if` clause simpler)

Something like:  (see what you think)
```python
        if k is not None and not endpoints:
            nodes = set(nodes)
            # Like above, we need to handle source nodes and non-source nodes
            # differently when using k while excluding endpoints.
            scale_source = N / ((K_source - 1) * correction) if K_source > 1 else float('nan')
            scale_nonsource = N / (K_source * correction)
            for v in betweenness:
                betweenness[v] *= scale_source if v in nodes else scale_nonsource
            return betweenness
        # no sampling adjustment needed
        scale = N / (K_source * correction)

    if scale != 1:
```
created_at: 2025-03-31T19:36:44Z
author_association: MEMBER
diff_hunk: @@ -128,10 +128,7 @@ def betweenness_centrality(
     """
     betweenness = dict.fromkeys(G, 0.0)  # b[v]=0 for v in G
     if k == len(G):
-        # This is for performance and correctness. When `endpoints` is False
-        # and k is given, k == n is a special case that would violate the
-        # assumption that node `v` is not one of the (s, t) node pairs.
-        # Should this warn or raise instead?
+        # This is done for performance; the result is the same regardless.
path: networkx/algorithms/centrality/betweenness.py
user: login: dschult
body: I'll go read ~~Blondel~~ Brandes again... Or maybe they wrote another paper more recently about directed vs undirected.
:)
created_at: 2025-03-31T20:45:30Z
author_association: MEMBER
diff_hunk: @@ -371,23 +371,50 @@ def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False
         # No rescaling necessary: b=0 for all nodes
         return betweenness
 
+    adjust_initial_samples = k is not None and nodes and not endpoints
+    if adjust_initial_samples:
+        nodes = set(nodes)
+
     K_source = N if k is None else k
     if normalized:
         # Divide by the number of valid (s, t) node pairs that could have
         # a path through v where s != t.
-        scale = 1 / (K_source * (N - 1))
+        if not adjust_initial_samples:
+            scale = 1 / (K_source * (N - 1))
+        else:
+            # Branch for excluding endpoints when using k. In this case, we need
+            # to handle source nodes differently from non-source nodes, because
+            # source nodes can't include themselves since endpoints are excluded.
+            # Without this, k == n would be a special case that would violate
+            # the assumption that node `v` is not one of the (s, t) node pairs.
+            # Avoid ZeroDivisionError when K_source == 1.
+            scale_source = 1 / ((K_source - 1) * (N - 1)) if K_source > 1 else 1
path: networkx/algorithms/centrality/betweenness.py
user: login: eriknw
body: Changed to nan :+1:
created_at: 2025-04-01T00:15:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -362,7 +360,9 @@ def _accumulate_edges(betweenness, S, P, sigma, s):
     return betweenness
 
 
-def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False):
+def _rescale(
+    betweenness, n, normalized, directed=False, k=None, endpoints=False, nodes=None
path: networkx/algorithms/centrality/betweenness.py
user: login: eriknw
body: Done :+1:
created_at: 2025-04-01T00:16:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -371,23 +371,50 @@ def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False
         # No rescaling necessary: b=0 for all nodes
         return betweenness
 
+    adjust_initial_samples = k is not None and nodes and not endpoints
path: networkx/algorithms/centrality/betweenness.py
user: login: eriknw
body: Done :+1:
created_at: 2025-04-01T00:16:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -371,23 +371,50 @@ def _rescale(betweenness, n, normalized, directed=False, k=None, endpoints=False
         # No rescaling necessary: b=0 for all nodes
         return betweenness
 
+    adjust_initial_samples = k is not None and nodes and not endpoints
+    if adjust_initial_samples:
+        nodes = set(nodes)
+
     K_source = N if k is None else k
     if normalized:
         # Divide by the number of valid (s, t) node pairs that could have
         # a path through v where s != t.
-        scale = 1 / (K_source * (N - 1))
+        if not adjust_initial_samples:
+            scale = 1 / (K_source * (N - 1))
+        else:
+            # Branch for excluding endpoints when using k. In this case, we need
+            # to handle source nodes differently from non-source nodes, because
+            # source nodes can't include themselves since endpoints are excluded.
+            # Without this, k == n would be a special case that would violate
+            # the assumption that node `v` is not one of the (s, t) node pairs.
+            # Avoid ZeroDivisionError when K_source == 1.
+            scale_source = 1 / ((K_source - 1) * (N - 1)) if K_source > 1 else 1
+            scale_nonsource = 1 / (K_source * (N - 1))
+            for v in betweenness:
+                betweenness[v] *= scale_source if v in nodes else scale_nonsource
     else:
         if not directed:
             # The non-normalized BC values are computed the same way for
             # directed and undirected graphs: shortest paths are computed and
             # counted for each *ordered* (s, t) pair. Undirected graphs should
             # only count valid *unordered* node pairs {s, t}; that is, (s, t)
             # and (t, s) should be counted only once. We correct for this here.
-            K_source *= 2
+            correction = 2
+        else:
+            correction = 1
         # Scale to the full BC
-        scale = N / K_source
-
-    if scale != 1:
+        if not adjust_initial_samples:
+            scale = N / (K_source * correction)
+        else:
+            # Like above, we need to handle source nodes and non-source nodes
+            # differently when using k while excluding endpoints.
+            # Avoid ZeroDivisionError when K_source == 1.
+            scale_source = N / ((K_source - 1) * correction) if K_source > 1 else 1
+            scale_nonsource = N / (K_source * correction)
+            for v in betweenness:
+                betweenness[v] *= scale_source if v in nodes else scale_nonsource
+
+    if not adjust_initial_samples and scale != 1:
path: networkx/algorithms/centrality/betweenness.py
user: login: eriknw
body: Thanks for the suggestions! As you saw, I took these suggestions and went a little further by doing "normal" case first, b/c it's easier to understand.
created_at: 2025-04-01T00:17:35Z
author_association: CONTRIBUTOR

PR comments:
user: login: eriknw
created_at: 2025-04-01T00:31:50Z
body: Thanks for the quick (and thorough, @dschult) reviews! I don't know of anything else that needs to be done before merging. I agree these changes are worth a detailed release note.

> Finally - I have one nit comment: should we use math.nan or float("nan")? I don't really know whether there's a difference and am happy with whatever we decide, but figured I'd raise it for my own edification!

Fair question! I looked into this too. `math.nan` and `float("nan")` are equivalent, and `math.nan` is guaranteed to always be available as of Python 3.11: https://docs.python.org/3/library/math.html#math.nan, which says of `math.nan` "Equivalent to the output of `float('nan')`".

So, the main differences are `math.nan` is shorter to type and is about 5x faster for me:
```python
In [2]: %timeit float("nan")
92.8 ns ± 5.64 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [3]: %timeit math.nan
18.6 ns ± 2.66 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
```

> **Undirected paths with k<n: [...]**

@dschult, believe it or not what you laid out actually makes sense to me, and some quick sympy-fu verified the results. It also matches my intuition :+1:.
author_association: CONTRIBUTOR
user: login: rossbar
created_at: 2025-04-01T02:41:22Z
body: Awesome - thanks again all!
author_association: CONTRIBUTOR