Repository: graspologic-org/graspologic

Issue:
bring oos stuff
- https://github.com/neurodata/primitives-interfaces/tree/master/jhu_primitives/oosase
- https://github.com/neurodata/primitives-interfaces/tree/master/jhu_primitives/ooslse
- https://github.com/neurodata/primitives-interfaces/tree/master/jhu_primitives/oosgmm

oosgmm could just be an option in gclust

the other two, it would be desirable to run some experiments evaluating the pro's/con's of sparse vs oos



Discussion:
@j1c and i talked about oosase and decided to add predict method to ase.

given adjacency matrix, A in (n, n)
and 
m out of sample adjacency vectors, X in (m, n):

ASE = AdjacenecySpectralEmbed()
in_sample_embedding = ASE.fit_transform(A)
X = (m, n) array 
oos_embedding = ASE.predict(X)
is there a paper or some more info associated with this?
i think this https://arxiv.org/pdf/1802.06307.pdf and I think we want the LLS version. some code linked above
really amounts to [this line](https://github.com/neurodata/primitives-interfaces/blob/d2773136a7dc30777c9b90c0dd8bcd0be12b167e/jhu_primitives/oosase/oosase.py#L248) which uses the pseudoinverse but I think is faster then `pinv` in numpy 
dope, I'm tentatively thinking I want to work on this
cool. IMO there's a lot of thinking to do w.r.t what interface makes the most sense here
old PR: #162
DoD:
- recreate figure 1 from oos paper to demonstrate successful implementation
- plan for API, discussed with @bdpedigo before implementation
- implementation of API PR'd
- tests (open to suggestions/changes)
    - is oos embedding almost always within an acceptable range of ground-truth embedding? (define "acceptable")
    - do independent OOS embeddings cluster harder around ground truth embedding as number of vertices increases?
    - does the least-squares implementation satisfy the moore-penrose conditions for pseudoinverse?
    - other tests will probably pop up as API is planned out
- tutorial notebook PRd into documentation
use cases (from @hhelm10 and @bdpedigo )
- an original graph is too big, so we want to make a signal subgraph and embed it, then find the latent positions for the rest of the nodes with oos
- we have a graph and embed it normally. Later, we get a new (maybe way bigger) graph that we're pretty sure was generated from the same underlying distribution, and we use oos embedding because it's easier computationally
- (maybe specific instance of second case) we have two graphs for which there is a partial one-to-one correspondence between vertices, and we know that partial correspondence -- but some vertices don't have known buddies. We omnibus embed a subgraph with only corresponding vertices, and use oos to embed the rest.

(any more?)

I'm thinking the best way to find an API that works well is to literally just simulate these use cases and play around with API implementation ideas until I'm able to do it in a way that feels simple and clean.

the other benefit of this approach is that the notebooks that come out of it could eventually become tutorials.
The third one I mentioned is probably not worth fretting about right now, though I do want it someday probably. But requires OOS Omni which I think is out of scope for this issue
@bdpedigo @hhelm10  can I get a better sense on here of what the problems you guys ran into re: clunkiness with the last iteration of the API in #162?
i think the clunkiness came from over-fitting the implementation to a specific application. if you are designing it and dont run into clunkiness concerns then i wouldnt be alarmed. 

Resolving patch:
diff --git a/graspologic/embed/ase.py b/graspologic/embed/ase.py
--- a/graspologic/embed/ase.py
+++ b/graspologic/embed/ase.py
@@ -2,6 +2,9 @@
 # Licensed under the MIT License.
 
 import warnings
+import numpy as np
+from sklearn.utils.validation import check_is_fitted
+import networkx as nx
 
 from .base import BaseSpectralEmbed
 from ..utils import (
@@ -103,6 +106,10 @@ class AdjacencySpectralEmbed(BaseSpectralEmbed):
     .. [1] Sussman, D.L., Tang, M., Fishkind, D.E., Priebe, C.E.  "A
        Consistent Adjacency Spectral Embedding for Stochastic Blockmodel Graphs,"
        Journal of the American Statistical Association, Vol. 107(499), 2012
+
+    .. [2] Levin, K., Roosta-Khorasani, F., Mahoney, M. W., & Priebe, C. E. (2018).
+        Out-of-sample extension of graph adjacency spectral embedding. PMLR: Proceedings
+        of Machine Learning Research, 80, 2975-2984.
     """
 
     def __init__(
@@ -127,6 +134,7 @@ def __init__(
         if not isinstance(diag_aug, bool):
             raise TypeError("`diag_aug` must be of type bool")
         self.diag_aug = diag_aug
+        self.is_fitted_ = False
 
     def fit(self, graph, y=None):
         """
@@ -137,6 +145,8 @@ def fit(self, graph, y=None):
         graph : array_like or networkx.Graph
             Input graph to embed.
 
+        y: Ignored
+
         Returns
         -------
         self : object
@@ -158,4 +168,95 @@ def fit(self, graph, y=None):
 
         self.n_features_in_ = len(A)
         self._reduce_dim(A)
+
+        # for out-of-sample
+        inv_eigs = np.diag(1 / self.singular_values_)
+        self._pinv_left = self.latent_left_ @ inv_eigs
+        if self.latent_right_ is not None:
+            self._pinv_right = self.latent_right_ @ inv_eigs
+
+        self.is_fitted_ = True
         return self
+
+    def transform(self, X):
+        """
+        Obtain latent positions from an adjacency matrix or matrix of out-of-sample
+        vertices. For more details on transforming out-of-sample vertices, see the
+        :ref:`tutorials <embed_tutorials>`. For mathematical background, see [2].
+
+        Parameters
+        ----------
+        X : array_like or tuple, original shape or (n_oos_vertices, n_vertices).
+
+            The original fitted matrix ("graph" in fit) or new out-of-sample data.
+            If ``X`` is the original fitted matrix, returns a matrix close to
+            ``self.fit_transform(X)``.
+
+            If ``X`` is an out-of-sample matrix, n_oos_vertices is the number
+            of new vertices, and n_vertices is the number of vertices in the
+            original graph. If tuple, graph is directed and ``X[0]`` contains
+            edges from out-of-sample vertices to in-sample vertices.
+
+        Returns
+        -------
+        array_like or tuple, shape (n_oos_vertices, n_components)
+        or (n_vertices, n_components).
+
+            Array of latent positions. Transforms the fitted matrix if it was passed
+            in.
+
+            If ``X`` is an array or tuple containing adjacency vectors corresponding to
+            new nodes, returns the estimated latent positions for the new out-of-sample
+            adjacency vectors.
+            If undirected, returns array.
+            If directed, returns ``(X_out, X_in)``, where ``X_out`` contains
+            latent positions corresponding to nodes with edges from out-of-sample
+            vertices to in-sample vertices.
+
+        Notes
+        -----
+        If the matrix was diagonally augmented (e.g., ``self.diag_aug`` was True), ``fit``
+        followed by ``transform`` will produce a slightly different matrix than
+        ``fit_transform``.
+
+        To get the original embedding, using ``fit_transform`` is recommended. In the
+        directed case, if A is the original in-sample adjacency matrix, the tuple
+        (A.T, A) will need to be passed to ``transform`` if you do not wish to use
+        ``fit_transform``.
+        """
+
+        # checks
+        check_is_fitted(self, "is_fitted_")
+        if isinstance(X, nx.classes.graph.Graph):
+            X = import_graph(X)
+        directed = self.latent_right_ is not None
+
+        # correct types?
+        if directed and not isinstance(X, tuple):
+            if X.shape[0] == X.shape[1]:  # in case original matrix was passed
+                msg = """A square matrix A was passed to ``transform`` in the directed case. 
+                If this was the original in-sample matrix, either use ``fit_transform`` 
+                or pass a tuple (A.T, A). If this was an out-of-sample matrix, directed
+                graphs require a tuple (X_out, X_in)."""
+                raise TypeError(msg)
+            else:
+                msg = "Directed graphs require a tuple (X_out, X_in) for out-of-sample transforms."
+                raise TypeError(msg)
+        if not directed and not isinstance(X, np.ndarray):
+            raise TypeError("Undirected graphs require array input")
+
+        # correct shape in y?
+        latent_rows = self.latent_left_.shape[0]
+        _X = X[0] if directed else X
+        X_cols = _X.shape[-1]
+        if _X.ndim > 2:
+            raise ValueError("out-of-sample vertex must be 1d or 2d")
+        if latent_rows != X_cols:
+            msg = "out-of-sample vertex must be shape (n_oos_vertices, n_vertices)"
+            raise ValueError(msg)
+
+        # workhorse code
+        if not directed:
+            return X @ self._pinv_left
+        elif directed:
+            return X[1] @ self._pinv_right, X[0] @ self._pinv_left
diff --git a/graspologic/utils/__init__.py b/graspologic/utils/__init__.py
--- a/graspologic/utils/__init__.py
+++ b/graspologic/utils/__init__.py
@@ -19,6 +19,7 @@
     binarize,
     cartprod,
     fit_plug_in_variance_estimator,
+    remove_vertices,
     remap_labels,
 )
 from .ptr import pass_to_ranks
@@ -42,5 +43,6 @@
     "cartprod",
     "pass_to_ranks",
     "fit_plug_in_variance_estimator",
+    "remove_vertices",
     "remap_labels",
 ]
diff --git a/graspologic/utils/utils.py b/graspologic/utils/utils.py
--- a/graspologic/utils/utils.py
+++ b/graspologic/utils/utils.py
@@ -708,6 +708,89 @@ def plug_in_variance_estimator(x):
     return plug_in_variance_estimator
 
 
+def remove_vertices(graph, indices, return_removed=False):
+    """
+    Remove a subgraph of adjacency vectors from an adjacency matrix, giving back the
+    truncated matrix and optionally the removed subgraph. Here, an adjacency vector
+    is the set of edge weights for a particular vertex.
+
+    Parameters
+    ----------
+    graph: networkx.Graph or array-like, shape (n, n)
+        The adjacency matrix for some graph.
+    indices: int or array-like, length m
+        Index/indices of the adjacency vector(s) to be removed.
+    return_removed: bool, by default False (optional)
+        Whether to return the tuple ``(A, V)``,
+        where ``A`` is the truncated adjacency matrix,
+        ``V`` is an array representing the removed subgraph.
+
+    Returns
+    -------
+    truncated_graph: np.ndarray
+        The truncated matrix.
+        This is a copy of `graph` of shape (k, k), with ``k=n-m``, without the ``m``
+        adjacency vectors given by `indices`.
+
+    removed_subgraph: np.ndarray or tuple, shape (m, k) (optional)
+        Array of removed adjacency vectors without edges to each other.
+        If directed, this is a tuple ``(V_1, V_2)``,
+        with ``V_1`` being an array of adjacency vectors from the removed subgraph to the truncated graph,
+        and ``V_2`` being an array of adjacency vectors from the truncated graph to the removed subgraph.
+
+    Examples
+    --------
+    # Undirected
+    >>> A = np.array([[0, 1, 2],
+                      [1, 0, 3],
+                      [2, 3, 0]])
+    >>> remove_vertices(A, 0)
+    array([[0., 3.],
+           [3., 0.]]))
+    >>> remove_vertices(A, 0, return_removed=True)
+    (array([[0., 3.],
+            [3., 0.]]),
+     array([1., 2.]))
+
+    # Directed
+    >>> B = np.array([[0, 1, 2, 3],
+                      [4, 0, 5, 6],
+                      [7, 8, 0, 9],
+                      [10, 11, 12, 0]])
+    >>> remove_vertices(B, 0, return_removed=True)
+    (array([[ 0.,  5.,  6.],
+            [ 8.,  0.,  9.],
+            [11., 12.,  0.]]),
+    (array([ 4.,  7., 10.]), array([1., 2., 3.])))
+    >>> remove_vertices(B, [0, -1], return_removed=True)
+    (array([[0., 5.],
+            [8., 0.]]),
+    (array([[4., 7.],
+            [6., 9.]]),
+    array([[ 1.,  2.],
+            [11., 12.]])))
+    """
+    graph = import_graph(graph)
+    if isinstance(indices, list) and len(indices) >= len(graph):
+        raise IndexError("You must pass in fewer vertex indices than vertices.")
+    directed = not is_almost_symmetric(graph)
+
+    # truncate graph
+    mask = np.ones(graph.shape[0], dtype=bool)
+    mask[indices] = 0
+    A = graph[mask, :][:, mask]
+
+    if return_removed:
+        rows = graph[mask]
+        vertices = rows[:, indices].T
+        if directed:
+            cols = graph[:, mask]
+            vertices_right = cols[indices]
+            return A, (vertices, vertices_right)
+        return A, vertices
+    return A
+
+
 def remap_labels(
     y_true: Union[List, np.ndarray, pd.Series],
     y_pred: Union[List, np.ndarray, pd.Series],


Test patch:
diff --git a/tests/test_sklearn.py b/tests/test_sklearn.py
--- a/tests/test_sklearn.py
+++ b/tests/test_sklearn.py
@@ -6,6 +6,5 @@
 
 from sklearn.utils.estimator_checks import check_estimator
 
-check_estimator(graspologic.embed.AdjacencySpectralEmbed)
 check_estimator(graspologic.embed.LaplacianSpectralEmbed)
 check_estimator(graspologic.embed.ClassicalMDS)
diff --git a/tests/test_spectral_embed.py b/tests/test_spectral_embed.py
--- a/tests/test_spectral_embed.py
+++ b/tests/test_spectral_embed.py
@@ -2,12 +2,21 @@
 # Licensed under the MIT License.
 
 import unittest
+import pytest
 import numpy as np
+import pandas as pd
+from numpy.random import poisson, normal
+from numpy.testing import assert_equal
+import networkx as nx
 from graspologic.embed.ase import AdjacencySpectralEmbed
 from graspologic.embed.lse import LaplacianSpectralEmbed
 from graspologic.simulations.simulations import er_np, er_nm, sbm
+from graspologic.utils import remove_vertices, is_symmetric
 from sklearn.cluster import KMeans
-from sklearn.metrics import adjusted_rand_score
+from sklearn.metrics import adjusted_rand_score, pairwise_distances
+from sklearn.base import clone
+
+np.random.seed(9002)
 
 
 def _kmeans_comparison(data, labels, n_clusters):
@@ -105,6 +114,23 @@ def _test_sbm_er_binary(self, method, P, directed=False, *args, **kwargs):
 
 
 class TestAdjacencySpectralEmbed(unittest.TestCase):
+    def setUp(self):
+        np.random.seed(9001)
+        n = [10, 10]
+        p = np.array([[0.9, 0.1], [0.1, 0.9]])
+        wt = [[normal, poisson], [poisson, normal]]
+        wtargs = [
+            [dict(loc=3, scale=1), dict(lam=5)],
+            [dict(lam=5), dict(loc=3, scale=1)],
+        ]
+        self.testgraphs = dict(
+            Guw=sbm(n=n, p=p),
+            Gw=sbm(n=n, p=p, wt=wt, wtargs=wtargs),
+            Guwd=sbm(n=n, p=p, directed=True),
+            Gwd=sbm(n=n, p=p, wt=wt, wtargs=wtargs, directed=True),
+        )
+        self.ase = AdjacencySpectralEmbed(n_components=2)
+
     def test_output_dim(self):
         _test_output_dim(self, AdjacencySpectralEmbed)
 
@@ -130,6 +156,109 @@ def test_input_checks(self):
             ase = AdjacencySpectralEmbed(diag_aug="over 9000")
             ase.fit()
 
+    def test_transform_closeto_fit_transform(self):
+        atol = 0.15
+        for diag_aug in [True, False]:
+            for g, A in self.testgraphs.items():
+                ase = AdjacencySpectralEmbed(n_components=2, diag_aug=diag_aug)
+                ase.fit(A)
+                Y = ase.fit_transform(A)
+                if isinstance(Y, np.ndarray):
+                    X = ase.transform(A)
+                    self.assertTrue(np.allclose(X, Y, atol=atol))
+                elif isinstance(Y, tuple):
+                    with self.assertRaises(TypeError):
+                        X = ase.transform(A)
+                    X = ase.transform((A.T, A))
+                    self.assertTrue(np.allclose(X[0], Y[0], atol=atol))
+                    self.assertTrue(np.allclose(X[1], Y[1], atol=atol))
+                else:
+                    raise TypeError
+
+    def test_transform_networkx(self):
+        G = nx.grid_2d_graph(5, 5)
+        ase = AdjacencySpectralEmbed(n_components=2)
+        ase.fit(G)
+        ase.transform(G)
+
+    def test_transform_correct_types(self):
+        ase = AdjacencySpectralEmbed(n_components=2)
+        for graph in self.testgraphs.values():
+            A, a = remove_vertices(graph, 1, return_removed=True)
+            ase.fit(A)
+            directed = ase.latent_right_ is not None
+            weighted = not np.array_equal(A, A.astype(bool))
+            w = ase.transform(a)
+            if directed:
+                self.assertIsInstance(w, tuple)
+                self.assertIsInstance(w[0], np.ndarray)
+                self.assertIsInstance(w[1], np.ndarray)
+            elif not directed:
+                self.assertIsInstance(w, np.ndarray)
+                self.assertEqual(np.atleast_2d(w).shape[1], 2)
+
+    def test_directed_vertex_direction(self):
+        M = self.testgraphs["Guwd"]
+        oos_idx = 0
+        A, a = remove_vertices(M, indices=oos_idx, return_removed=True)
+        assert_equal(np.delete(M[:, 0], oos_idx), a[0])
+
+    def test_directed_correct_latent_positions(self):
+        # setup
+        ase = AdjacencySpectralEmbed(n_components=3)
+        P = np.array([[0.9, 0.1, 0.1], [0.3, 0.6, 0.1], [0.1, 0.5, 0.6]])
+        M, labels = sbm([200, 200, 200], P, directed=True, return_labels=True)
+
+        # one node from each community
+        oos_idx = np.nonzero(np.r_[1, np.diff(labels)[:-1]])[0]
+        labels = list(labels)
+        oos_labels = [labels.pop(i) for i in oos_idx]
+
+        # Grab out-of-sample, fit, transform
+        A, a = remove_vertices(M, indices=oos_idx, return_removed=True)
+        latent_left, latent_right = ase.fit_transform(A)
+        oos_left, oos_right = ase.transform(a)
+
+        # separate into communities
+        for i, latent in enumerate([latent_left, latent_right]):
+            left = i == 0
+            df = pd.DataFrame(
+                {
+                    "Type": labels,
+                    "Dimension 1": latent[:, 0],
+                    "Dimension 2": latent[:, 1],
+                    "Dimension 3": latent[:, 2],
+                }
+            )
+            # make sure that oos vertices are closer to their true community averages than other community averages
+            means = df.groupby("Type").mean()
+            if left:
+                avg_dist_within = np.diag(pairwise_distances(means, oos_left))
+                avg_dist_between = np.diag(pairwise_distances(means, oos_right))
+                self.assertTrue(all(avg_dist_within < avg_dist_between))
+            elif not left:
+                avg_dist_within = np.diag(pairwise_distances(means, oos_right))
+                avg_dist_between = np.diag(pairwise_distances(means, oos_left))
+                self.assertTrue(all(avg_dist_within < avg_dist_between))
+
+    def test_exceptions(self):
+        ase = clone(self.ase)
+
+        with pytest.raises(Exception):
+            ase.fit(self.testgraphs["Gwd"])
+            ase.transform("9001")
+
+        with pytest.raises(Exception):
+            Guwd = self.testgraphs["Guwd"]
+            ase.fit(Guwd)
+            ase.transform(np.ones(len(Guwd)))
+
+        with pytest.raises(ValueError):
+            A, a = remove_vertices(self.testgraphs["Gw"], [0, 1], return_removed=True)
+            a = a.T
+            ase.fit(A)
+            ase.transform(a)
+
 
 class TestLaplacianSpectralEmbed(unittest.TestCase):
     def test_output_dim(self):
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -2,6 +2,10 @@
 # Licensed under the MIT License.
 
 import unittest
+import pytest
+import numpy as np
+import networkx as nx
+from graspologic.utils import utils as gus
 from math import sqrt
 
 import networkx as nx
@@ -418,6 +422,90 @@ def test_binarize():
     assert_equal(g2, g2_expected)
 
 
+class TestRemoveVertices(unittest.TestCase):
+    def setUp(self):
+        self.directed = np.array(
+            [
+                [0, 2, 3, 4, 5],
+                [6, 0, 8, 9, 10],
+                [11, 12, 0, 14, 15],
+                [16, 17, 18, 0, 20],
+                [21, 22, 23, 24, 0],
+            ]
+        )
+        self.undirected = np.array(
+            [
+                [0, 6, 11, 16, 21],
+                [6, 0, 12, 17, 22],
+                [11, 12, 0, 18, 23],
+                [16, 17, 18, 0, 24],
+                [21, 22, 23, 24, 0],
+            ]
+        )
+
+    def test_undirected(self):
+        # with list index
+        indices = [0, -1, 1]
+        for idx in [indices, np.array(indices)]:
+            A, a = gus.remove_vertices(self.undirected, idx, return_removed=True)
+            self.assertIsInstance(a, np.ndarray)
+            assert_equal(A, np.array([[0, 18], [18, 0]]))
+            assert_equal(a, np.array([[11, 16], [23, 24], [12, 17]]))
+            self.assertTrue(gus.is_almost_symmetric(A))
+
+        # with integer index
+        indices = 0
+        A, a = gus.remove_vertices(self.undirected, indices, return_removed=True)
+        assert_equal(A, self.undirected[1:, 1:])
+        assert_equal(a, np.array([6, 11, 16, 21]))
+        self.assertTrue(gus.is_almost_symmetric(A))
+        assert_equal(
+            gus.remove_vertices(self.undirected, 0),
+            gus.remove_vertices(self.undirected, [0]),
+        )
+
+    def test_directed(self):
+        # with list index
+        idx1 = [0, -1, 1]
+        idx2 = np.array(idx1)
+        for indices in [idx1, idx2]:
+            A, a = gus.remove_vertices(self.directed, indices, return_removed=True)
+            self.assertIsInstance(a, tuple)
+            self.assertIsInstance(a[0], np.ndarray)
+            self.assertIsInstance(a[1], np.ndarray)
+            assert_equal(A, np.array([[0, 14], [18, 0]]))
+            assert_equal(a[0], np.array([[11, 16], [15, 20], [12, 17]]))
+            assert_equal(a[1], np.array([[3, 4], [23, 24], [8, 9]]))
+
+        # with integer index
+        idx = 0
+        A, a = gus.remove_vertices(self.directed, idx, return_removed=True)
+        assert_equal(A, gus.remove_vertices(self.directed, idx))
+        self.assertIsInstance(a, tuple)
+        self.assertIsInstance(a[0], np.ndarray)
+        self.assertIsInstance(a[1], np.ndarray)
+        assert_equal(A, self.directed[1:, 1:])
+        assert_equal(a[0], np.array([6, 11, 16, 21]))
+        assert_equal(a[1], np.array([2, 3, 4, 5]))
+
+    def test_exceptions(self):
+        # ensure proper errors are thrown when invalid inputs are passed.
+        with pytest.raises(TypeError):
+            gus.remove_vertices(9001, 0)
+
+        with pytest.raises(ValueError):
+            nonsquare = np.vstack((self.directed, self.directed))
+            gus.remove_vertices(nonsquare, 0)
+
+        with pytest.raises(IndexError):
+            indices = np.arange(len(self.directed) + 1)
+            gus.remove_vertices(self.directed, indices)
+
+        with pytest.raises(IndexError):
+            idx = len(self.directed) + 1
+            gus.remove_vertices(self.directed, indices)
+
+
 class TestRemapLabels(unittest.TestCase):
     @classmethod
     def setUpClass(cls):