Repository: chanzuckerberg/single-cell-curation

Issue:
Cannot validate large h5ad files because the entire X matrix is read into memory
## Motivation
This may just be a question (though, depending on the answer, it might be a feature request).

I'm a scientist at the Allen Institute for Brain Science. We have a single cell rnaseq dataset comprised of 4 million cells and 30 thousand genes. Stored as a sparse matrix in an h5ad file, this produces a 215 GB file. We would like to experiment with aligning our files with the cellxgene curation schema. However, running the validation tool is impossible. This line of code

https://github.com/chanzuckerberg/single-cell-curation/blob/main/cellxgene_schema_cli/cellxgene_schema/validate.py#L111-L113

means that the first thing the validation tool tries to do is read the full 215 GB X matrix into memory. I don't know the history of this choice. Is there no work around that will enable the h5ad to be read in with `backed='r'`?


Discussion:
Hi @danielsf - this sounds fantastic! I have also been experimenting with very large datasets. The `backed` feature solves some issues, but is not a panacea if the X data is needed (the validator currently requires part of X, to perform a spot validation). Could you share a bit more info to help us identify possible paths forward?

1. What is a reasonable memory budget in your computing environment? In other words, what is the available RAM on the machine you are running the validator?
2. Is your goal submission of the dataset to our CELLxGENE portal, or just to explore other uses of the validator?  If other, would you be open to sharing your goal?

Thanks!
Hi @bkmartinjr 

1) It would be nice if I could run this on a machine with 64 GB of memory. I could come by a 128 GB machine if I needed to, though.

2) We're still very much in the exploratory phase. At this point, we have this dataset, we want *some* tool to validate it, and we thought maybe we could leverage pre-existing tools. Whether or not the data ultimately gets submitted to CELLxGENE is still an open question (though the answer is not "no"). Sorry to be so vague.

Is there anyway to use anndata's chunking mechanism to do that spot check? That might limit the memory requirements.
There are definitely some possible avenues to reducing memory footprint (possibly at the cost of time). I will do a bit of investigation and see what turns up.  I don't think chunking will help per se (if you look at how it is implemented, it just uses the standard indexer on X).

One more question: what is the format / type of your X matrix?  CSR or something else?

Resolving patch:
diff --git a/cellxgene_schema_cli/cellxgene_schema/validate.py b/cellxgene_schema_cli/cellxgene_schema/validate.py
--- a/cellxgene_schema_cli/cellxgene_schema/validate.py
+++ b/cellxgene_schema_cli/cellxgene_schema/validate.py
@@ -1,4 +1,5 @@
 import logging
+import math
 import re
 import sys
 from datetime import datetime
@@ -6,15 +7,10 @@
 import anndata
 import os
 import pandas as pd
+import numpy as np
 from pandas.core.computation.ops import UndefinedVariableError
-from numpy import ndarray
-from numpy import count_nonzero
-from numpy import cumprod
-from numpy import isnan
-from numpy import nditer
-from numpy import vectorize
 from scipy import sparse
-from typing import List, Dict, Union, Tuple
+from typing import List, Dict, Union, Tuple, Optional
 from . import ontology
 from . import schema
 from . import env
@@ -30,7 +26,6 @@ class Validator:
     schema_definitions_dir = env.SCHEMA_DEFINITIONS_DIR
 
     def __init__(self, ignore_labels=False):
-
         # Set initial state
         self.errors = []
         self.warnings = []
@@ -47,6 +42,9 @@ def __init__(self, ignore_labels=False):
         # keys will be one of ontology.SupportedOrganisms
         self.gene_checkers = dict()
 
+        # Matrix (e.g., X, raw.X, ...) number non-zero cache
+        self.number_non_zero = dict()
+
     @staticmethod
     def _curie_remove_suffix(term_id: str, suffix_def: dict) -> Tuple[str, str]:
         """
@@ -63,7 +61,6 @@ def _curie_remove_suffix(term_id: str, suffix_def: dict) -> Tuple[str, str]:
         id_suffix = ""
 
         for ontology_name, suffixes in suffix_def.items():
-
             for suffix in suffixes:
                 suffix = suffix.replace("(", r"\(")
                 suffix = suffix.replace(")", r"\)")
@@ -81,7 +78,6 @@ def _curie_remove_suffix(term_id: str, suffix_def: dict) -> Tuple[str, str]:
 
     @staticmethod
     def getattr_anndata(adata: anndata.AnnData, attr: str = None):
-
         """
         same as getattr but handles the special case of "raw.var" for an anndata.AndData object
 
@@ -100,7 +96,6 @@ def getattr_anndata(adata: anndata.AnnData, attr: str = None):
             return getattr(adata, attr)
 
     def _read_h5ad(self, h5ad_path: Union[str, bytes, os.PathLike]):
-
         """
         Reads h5ad into self.adata
         :params Union[str, bytes, os.PathLike] h5ad_path: path to h5ad to read
@@ -108,9 +103,20 @@ def _read_h5ad(self, h5ad_path: Union[str, bytes, os.PathLike]):
         :rtype None
         """
         try:
-            # H5AD has to be loaded in memory mode. If not the types of X are not properly retrieved by anndata
-            # see https://github.com/theislab/anndata/issues/326#issuecomment-892203924
-            self.adata = anndata.read_h5ad(h5ad_path, backed=None)
+            self.adata = anndata.read_h5ad(h5ad_path, backed="r")
+
+            # This code, and AnnData in general, is optimized for row access.
+            # Running backed, with CSC, is prohibitively slow. Read the entire
+            # AnnData into memory if it is CSC.
+            if (self._get_matrix_format(self.adata.X) == "csc") or (
+                (self.adata.raw is not None)
+                and (self._get_matrix_format(self.adata.raw.X) == "csc")
+            ):
+                logger.warning(
+                    "Matrices are in CSC format; loading entire dataset into memory."
+                )
+                self.adata = self.adata.to_memory()
+
         except (OSError, TypeError):
             logger.info(f"Unable to open '{h5ad_path}' with AnnData")
             sys.exit(1)
@@ -138,7 +144,6 @@ def _set_schema_def(self):
                 "adata has no schema definition in 'adata.uns'. Validation cannot be performed."
             )
         else:
-
             # Check if schema version is supported
             version = self.adata.uns["schema_version"]
             path = schema.get_schema_file_path(version)
@@ -185,7 +190,6 @@ def _get_column_def(self, component: str, column_name: str) -> dict:
     def _validate_curie_allowed_terms(
         self, term_id: str, column_name: str, terms: Dict[str, List[str]]
     ):
-
         """
         Validate a single curie term id is a valid children of any of allowed terms
         If there are any errors, it adds them to self.errors
@@ -216,7 +220,6 @@ def _validate_curie_ancestors(
         inclusive: bool,
         errors=True,
     ):
-
         """
         Validate a single curie term id is a valid children of any of allowed ancestors
         If there are any errors, it adds them to self.errors if the errors flag is True.
@@ -235,7 +238,6 @@ def _validate_curie_ancestors(
 
         for ontology_name, ancestors in allowed_ancestors.items():
             for ancestor in ancestors:
-
                 if inclusive and term_id == ancestor:
                     checks.append(True)
 
@@ -263,7 +265,6 @@ def _validate_curie_ancestors(
     def _validate_curie_ontology(
         self, term_id: str, column_name: str, allowed_ontologies: List[str]
     ):
-
         """
         Validate a single curie term id belongs to specified ontologies. If it does belong to an allowed ontology
         verifies that it is not deprecated (obsolete).
@@ -279,7 +280,6 @@ def _validate_curie_ontology(
         checks = []
 
         for ontology_name in allowed_ontologies:
-
             is_valid = ONTOLOGY_CHECKER.is_valid_term_id(ontology_name, term_id)
             checks.append(is_valid)
 
@@ -295,7 +295,6 @@ def _validate_curie_ontology(
             )
 
     def _validate_curie(self, term_id: str, column_name: str, curie_constraints: dict):
-
         """
         Validate a single curie term id based on some constraints.
         If there are any errors, it adds them to self.errors
@@ -361,7 +360,6 @@ def _validate_curie(self, term_id: str, column_name: str, curie_constraints: dic
             )
 
     def _validate_feature_id(self, feature_id: str, df_name: str):
-
         """
         Validates a feature id, i.e. checks that it's present in the reference
         If there are any errors, it adds them to self.errors and adds it to the list of invalid features
@@ -392,10 +390,82 @@ def _validate_feature_id(self, feature_id: str, df_name: str):
 
         return
 
+    def _get_matrix_format(self, matrix: Union[np.ndarray, sparse.spmatrix]) -> str:
+        """
+        Given a matrix, returns the format as one of: csc, csr, coo, dense
+        or unknown.
+
+        This mimics the scipy.sparse `format` property, but extends it to
+        support ndarray and other classes AnnData may proxy the matrix with.
+        """
+
+        # Note: the AnnData proxy classes DO support the `format_str` property, but
+        # doing a slice seemed safer, if less performant.  Using `format_str`, which
+        # currently works, uses private API:
+        #
+        # >>> return getattr(matrix, "format_str", "dense)
+        #
+        format = "unknown"
+        if self.adata.n_obs == 0 or self.adata.n_vars == 0:
+            format = "dense"
+        else:
+            matrix_slice = matrix[0:1, 0:1]
+            if isinstance(matrix_slice, sparse.spmatrix):
+                format = matrix_slice.format
+            elif isinstance(matrix_slice, np.ndarray):
+                format = "dense"
+
+        assert format in ["unknown", "csr", "csc", "coo", "dense"]
+        return format
+
+    def _chunk_matrix(
+        self,
+        matrix: Union[np.ndarray, sparse.spmatrix],
+        obs_chunk_size: Optional[int] = 10_000,
+    ):
+        """
+        Iterator which chunks the _named_ or _specified_ matrix by the
+        first (obs) dimension
+
+        The parameter type restrictions are strictly for ensuring that the
+        AnnData read fast-path is used (as of AnnData 0.8.0).
+
+        Iterator produces a sequence of tuples, each containing
+        (chunk, start, end)
+        """
+        start = 0
+        n = matrix.shape[0]
+        for i in range(int(n // obs_chunk_size)):
+            logger.debug(f"_chunk_matrix [{i} of {math.ceil(n/obs_chunk_size)}]")
+            end = start + obs_chunk_size
+            yield (matrix[start:end], start, end)
+            start = end
+        if start < n:
+            yield (matrix[start:n], start, n)
+
+    def _count_matrix_nonzero(
+        self, matrix_name: str, matrix: Union[np.ndarray, sparse.spmatrix]
+    ) -> int:
+        if matrix_name in self.number_non_zero:
+            return self.number_non_zero[matrix_name]
+
+        logger.debug(f"Counting non-zero values in {matrix_name}")
+
+        nnz = 0
+        format = self._get_matrix_format(matrix)
+        for matrix_chunk, _, _ in self._chunk_matrix(matrix):
+            nnz += (
+                matrix_chunk.count_nonzero()
+                if format != "dense"
+                else np.count_nonzero(matrix_chunk)
+            )
+
+        self.number_non_zero[matrix_name] = nnz
+        return nnz
+
     def _validate_column_feature_is_filtered(
         self, column: pd.Series, column_name: str, df_name: str
     ):
-
         """
         Validates the "is_feature_filtered" in adata.var. This column must be bool, and for genes that are set to
         True, their expression values in X must be 0.
@@ -411,19 +481,14 @@ def _validate_column_feature_is_filtered(
             return
 
         if sum(column) > 0:
-
             n_nonzero = 0
 
-            if (
-                isinstance(self.adata.X, sparse.csc_matrix)
-                or isinstance(self.adata.X, sparse.csr_matrix)
-                or isinstance(self.adata.X, sparse.coo_matrix)
-            ):
-
+            X_format = self._get_matrix_format(self.adata.X)
+            if X_format in ["csc", "csr", "coo"]:
                 n_nonzero = self.adata.X[:, column].count_nonzero()
 
-            elif isinstance(self.adata.X, ndarray):
-                n_nonzero = count_nonzero(self.adata.X[:, column])
+            elif X_format == "dense":
+                n_nonzero = np.count_nonzero(self.adata.X[:, column])
 
             else:
                 self.errors.append(
@@ -441,7 +506,6 @@ def _validate_column_feature_is_filtered(
     def _validate_column(
         self, column: pd.Series, column_name: str, df_name: str, column_def: dict
     ):
-
         """
         Given a schema definition and the column of a dataframe, verify that the column satisfies the schema.
         If there are any errors, it adds them to self.errors
@@ -514,13 +578,11 @@ def _validate_column(
                 )
 
         if column_def.get("type") == "feature_id":
-
             # Validates each id
             for feature_id in column:
                 self._validate_feature_id(feature_id, df_name)
 
         if column_def.get("type") == "curie":
-
             # Check for NaN values
             if column.isnull().any():
                 self.errors.append(
@@ -553,7 +615,6 @@ def _validate_column(
     def _validate_column_dependencies(
         self, df: pd.DataFrame, df_name: str, column_name: str, dependencies: List[dict]
     ) -> pd.Series:
-
         """
         Validates subset of columns based on dependecies, for instance development_stage_ontology_term_id has
         dependencies with organism_ontology_term_id -- the allowed values depend on whether organism is human, mouse
@@ -590,7 +651,7 @@ def _validate_column_dependencies(
                     f"Checking values with dependencies failed for adata.{df_name}['{column_name}'], "
                     f"this is likely due to missing dependent column in adata.{df_name}."
                 )
-                return pd.Series()
+                return pd.Series(dtype=np.float64)
 
             all_rules.append(query_exp)
 
@@ -611,7 +672,9 @@ def _generate_match_ancestors_query_fn(self, rule_def: Dict):
         :param rule_def: defines arguments to pass into vectorized ancestor match validation function
         :return: Tuple(function, Tuple(str, List[str], List[str]))
         """
-        validate_curie_ancestors_vectorized = vectorize(self._validate_curie_ancestors)
+        validate_curie_ancestors_vectorized = np.vectorize(
+            self._validate_curie_ancestors
+        )
         ancestor_map = rule_def["ancestors"]
         inclusive = rule_def["inclusive"]
 
@@ -643,7 +706,6 @@ def is_ancestor_match(
     def _validate_list(
         self, list_name: str, current_list: List[str], element_type: str
     ):
-
         """
         Validates the elements of a list based on the type definition. Adds errors to self.errors if any
 
@@ -655,7 +717,6 @@ def _validate_list(
         """
 
         for i in current_list:
-
             if element_type == "match_obs_columns":
                 if i not in self.adata.obs.columns:
                     self.errors.append(
@@ -663,7 +724,6 @@ def _validate_list(
                     )
 
     def _validate_str_in_dict(self, value, dict_name: str, key: str):
-
         """
         Validates that a value from a dictionary is a string and it does not have leading, trailing or double spaces.
         Adds errors to self.errors if any
@@ -685,7 +745,6 @@ def _validate_str_in_dict(self, value, dict_name: str, key: str):
 
             is_valid = False
         else:
-
             if value != value.rstrip():
                 self.errors.append(
                     f"'{value}' in '{dict_name}['{key}']' is not valid, it contains trailing spaces."
@@ -710,7 +769,6 @@ def _validate_str_in_dict(self, value, dict_name: str, key: str):
         return is_valid
 
     def _validate_enum_in_dict(self, value, enum: List[str], dict_name: str, key: str):
-
         """
         Validates that a value from a dictionary is part of a list. Adds errors to self.errors if any
 
@@ -729,7 +787,6 @@ def _validate_enum_in_dict(self, value, enum: List[str], dict_name: str, key: st
             )
 
     def _validate_dict(self, dictionary: dict, dict_name: str, dict_def: dict):
-
         """
         Verifies the dictionary follows the schema. Adds errors to self.errors if any
 
@@ -759,7 +816,6 @@ def _validate_dict(self, dictionary: dict, dict_name: str, dict_def: dict):
                     )
 
             if value_def["type"] == "match_obsm_keys":
-
                 if not self._validate_str_in_dict(value, dict_name, key):
                     continue
 
@@ -770,7 +826,7 @@ def _validate_dict(self, dictionary: dict, dict_name: str, dict_def: dict):
                     )
 
             if value_def["type"] == "list":
-                if not (isinstance(value, list) or isinstance(value, ndarray)):
+                if not (isinstance(value, list) or isinstance(value, np.ndarray)):
                     self.errors.append(
                         f"'{value}' in '{dict_name}['{key}']' is not valid, "
                         f"it must be a list or numpy array."
@@ -780,7 +836,6 @@ def _validate_dict(self, dictionary: dict, dict_name: str, dict_def: dict):
                 self._validate_list(key, value, value_def["element_type"])
 
     def _validate_dataframe(self, df_name: str):
-
         """
         Verifies the dataframe follows the schema. Adds errors to self.errors if any
 
@@ -852,51 +907,47 @@ def _validate_dataframe(self, df_name: str):
                     self._validate_column(column, column_name, df_name, column_def)
 
     def _validate_sparsity(self):
-
         """
         calculates sparsity of x and raw.x, if bigger than indicated in the schema and not a scipy sparse matrix, then
         adds to warnings
 
         :rtype none
         """
-
         max_sparsity = float(self.schema_def["sparsity"])
 
-        to_validate = [self.adata.X]
-        to_validate_name = ["X"]
+        to_validate = [(self.adata.X, "X")]
 
         # check if there's raw data
         if self.adata.raw:
-            to_validate.append(self.adata.raw.X)
-            to_validate_name.append("raw")
+            to_validate.append((self.adata.raw.X, "raw.X"))
 
         # check if there's other expression matrices under layers
         if self.adata.layers:
             for key, value in self.adata.layers.items():
-                to_validate.append(value)
-                to_validate_name.append(f"layers['{key}']")
+                to_validate.append((value, f"layers['{key}']"))
 
         # Check sparsity
-        for x, x_name in zip(to_validate, to_validate_name):
-            if not isinstance(x, sparse.csr_matrix):
-
-                if isinstance(x, sparse.csc_matrix) or isinstance(x, sparse.coo_matrix):
-                    sparsity = 1 - x.count_nonzero() / float(cumprod(x.shape)[-1])
-                elif isinstance(x, ndarray):
-                    sparsity = 1 - count_nonzero(x) / float(cumprod(x.shape)[-1])
-                else:
-                    self.warnings.append(
-                        f"{x_name} matrix is of type {type(x)}, sparsity calculation hasn't been "
-                        f"implemented"
-                    )
-                    continue
-
-                if sparsity > max_sparsity:
-                    self.warnings.append(
-                        f"Sparsity of '{x_name}' is {sparsity} which is greater than {max_sparsity}, "
-                        f"and it is not a 'scipy.sparse.csr_matrix'. It is STRONGLY RECOMMENDED "
-                        f"to use this type of matrix for the given sparsity."
-                    )
+        for x, x_name in to_validate:
+            matrix_format = self._get_matrix_format(x)
+            if matrix_format == "csr":
+                continue
+            assert format != "unknown"
+
+            # It seems silly to perform this test for 'coo' and 'csc' formats,
+            # which are, by definition, already sparse. But the old code
+            # performs test, and so we continue the tradition. It is possible
+            # that the prolog comment is incorrect, and the purpose of this
+            # function is to recommend CSR for _any_ matrix with sparsity beyond
+            # a given limit.
+
+            nnz = self._count_matrix_nonzero(x_name, x)
+            sparsity = 1 - nnz / np.prod(x.shape)
+            if sparsity > max_sparsity:
+                self.warnings.append(
+                    f"Sparsity of '{x_name}' is {sparsity} which is greater than {max_sparsity}, "
+                    f"and it is not a 'scipy.sparse.csr_matrix'. It is STRONGLY RECOMMENDED "
+                    f"to use this type of matrix for the given sparsity."
+                )
 
     def _validate_seurat_convertibility(self):
         """
@@ -905,18 +956,17 @@ def _validate_seurat_convertibility(self):
         too large.
         rtype: None
         """
-        to_validate = [self.adata.X]
-        to_validate_name = ["X"]
+        to_validate = [(self.adata.X, "X")]
         # check if there's raw data
         if self.adata.raw:
-            to_validate.append(self.adata.raw.X)
-            to_validate_name.append("raw")
+            to_validate.append((self.adata.raw.X, "raw.X"))
         # Check length of component arrays
-        for matrix, matrix_name in zip(to_validate, to_validate_name):
-            if sparse.issparse(matrix):
-                effective_r_array_size = matrix.count_nonzero()
+        for matrix, matrix_name in to_validate:
+            format = self._get_matrix_format(matrix)
+            if format in ["csc", "csr", "coo"]:
+                effective_r_array_size = self._count_matrix_nonzero(matrix_name, matrix)
                 is_sparse = True
-            elif isinstance(matrix, ndarray):
+            elif format == "dense":
                 effective_r_array_size = max(matrix.shape)
                 is_sparse = False
             else:
@@ -925,6 +975,7 @@ def _validate_seurat_convertibility(self):
                     f"of type {type(matrix)}"
                 )
                 continue
+
             if effective_r_array_size > self.schema_def["max_size_for_seurat"]:
                 if is_sparse:
                     self.warnings.append(
@@ -944,7 +995,6 @@ def _validate_seurat_convertibility(self):
                 self.is_seurat_convertible = False
 
     def _validate_embedding_dict(self):
-
         """
         Validates the embedding dictionary -- it checks that all values of adata.obms are numpy arrays with the correct
         dimension. Adds errors to self.errors if any. Checks that the keys start with "X_"
@@ -958,8 +1008,7 @@ def _validate_embedding_dict(self):
 
         obsm_with_x_prefix = 0
         for key, value in self.adata.obsm.items():
-
-            if not isinstance(value, ndarray):
+            if not isinstance(value, np.ndarray):
                 self.errors.append(
                     f"All embeddings have to be of 'numpy.ndarray' type, "
                     f"'adata.obsm['{key}']' is {type(value)}')."
@@ -990,7 +1039,6 @@ def _validate_embedding_dict(self):
     def _are_children_of(
         self, component: str, column: str, ontology_name: str, ancestors: List[str]
     ) -> bool:
-
         """
         Checks if elements in the specified column of the component (e.g. 'assay_ontology_term_id' of 'adata.obs') are
         children of the given ancestors.
@@ -1020,8 +1068,7 @@ def _are_children_of(
 
         return False
 
-    def _get_raw_x(self) -> Union[ndarray, sparse.csc_matrix, sparse.csr_matrix]:
-
+    def _get_raw_x(self) -> Union[np.ndarray, sparse.csc_matrix, sparse.csr_matrix]:
         """
         gets raw x (best guess, i.e. not guarantee it's actually raw)
         """
@@ -1032,7 +1079,6 @@ def _get_raw_x(self) -> Union[ndarray, sparse.csc_matrix, sparse.csr_matrix]:
             return self.adata.X
 
     def _get_raw_x_loc(self) -> str:
-
         """
         gets raw x location (best guess, i.e. not guarantee it's actually raw)
         """
@@ -1043,7 +1089,6 @@ def _get_raw_x_loc(self) -> str:
             return "X"
 
     def _is_raw(self, max_values_to_check: int = 5000, force: bool = False) -> bool:
-
         """
         Checks if the first non-zero "max_values_to_check" in the best guess for the raw matrix (adata.X or adata.raw.X)
         are integers. Returns False if at least one value is not an integer,
@@ -1058,12 +1103,10 @@ def _is_raw(self, max_values_to_check: int = 5000, force: bool = False) -> bool:
         :rtype bool
         :return False if at least one value is not an integer, True otherwise
         """
-
         if force:
             self._raw_layer_exists = None
 
         if self._raw_layer_exists is None:
-
             # Get potential raw_X
             raw_loc = self._get_raw_x_loc()
             if raw_loc == "raw.X":
@@ -1071,36 +1114,31 @@ def _is_raw(self, max_values_to_check: int = 5000, force: bool = False) -> bool:
             else:
                 x = self.adata.X
 
-            # Get array without zeros
-            logger.debug("Getting nonzero matrix for type (int) validation...")
-            start = datetime.now()
-            non_zeroes_index = x.nonzero()
-            if max_values_to_check > len(non_zeroes_index[0]):
-                max_values_to_check = len(non_zeroes_index[0])
-
-            x_non_zeroes = x[
-                non_zeroes_index[0][:max_values_to_check],
-                non_zeroes_index[1][:max_values_to_check],
-            ]
-            logger.debug(f"Copy of nonzero matrix created in: {datetime.now() - start}")
-            # If all values are zeros then is raw, otherwise if a single value is not an int then return is not raw
-            if x_non_zeroes.size < 1:
-                self._raw_layer_exists = True
-            else:
-                self._raw_layer_exists = True
-                for i in nditer(x_non_zeroes):
-
-                    if isnan(i):
-                        continue
-
-                    if i % 1 != 0:
-                        self._raw_layer_exists = False
-                        break
+            num_values_checked = 0
+            format = self._get_matrix_format(x)
+            assert format != "unknown"
+            self._raw_layer_exists = True
+            for matrix_chunk, _, _ in self._chunk_matrix(x):
+                data = (
+                    matrix_chunk
+                    if isinstance(matrix_chunk, np.ndarray)
+                    else matrix_chunk.data
+                )
+                if (data % 1 > 0).any():
+                    self._raw_layer_exists = False
+                    break
+
+                num_values_checked += (
+                    matrix_chunk.nnz
+                    if format != "dense"
+                    else np.count_nonzero(matrix_chunk)
+                )
+                if num_values_checked > max_values_to_check:
+                    break
 
         return self._raw_layer_exists
 
     def _validate_x_raw_x_dimensions(self):
-
         """
         Validates that X and raw.X have the same shape. Adds errors to self.errors if any.
         """
@@ -1126,7 +1164,6 @@ def _validate_x_raw_x_dimensions(self):
                     self.errors.append("Cells in X and raw.X are different.")
 
     def _validate_raw(self):
-
         """
         Validates raw only if the rules in the schema definition are fulfilled and that X and raw.X have the same shape
         The validation entails checking that:
@@ -1162,7 +1199,6 @@ def _validate_raw(self):
 
         # If all checks passed then proceed with validation
         if all(checks):
-
             # If both "raw.X" and "X" exist but neither are raw
             # This is testing for when sometimes data contributors put a normalized matrix in both "X" and "raw.X".
             if not self._is_raw() and self._get_raw_x_loc() == "raw.X":
@@ -1188,7 +1224,6 @@ def _validate_raw(self):
     def _check_single_column_availability(
         self, component: str, add_labels_def: List[dict]
     ):
-
         """
         This method checks a single reserved column in adata.obs or adata.var and adds a message to self.error if
         it already exists
@@ -1241,7 +1276,6 @@ def _check_invalid_columns(self):
                     )
 
     def _check_column_availability(self):
-
         """
         This method will check for columns that are reserved in self.adata.obs or
         self.adata.var already exist
@@ -1250,7 +1284,6 @@ def _check_column_availability(self):
         """
 
         for component, component_def in self.schema_def["components"].items():
-
             # If not a dataframe we don't need to check for columns
             if component_def["type"] != "dataframe":
                 continue
@@ -1274,7 +1307,6 @@ def _check_column_availability(self):
             # Do it for columns that map to columns
             if "columns" in component_def:
                 for column, columns_def in component_def["columns"].items():
-
                     if "add_labels" in columns_def:
                         self._check_single_column_availability(
                             component, columns_def["add_labels"]
@@ -1282,7 +1314,6 @@ def _check_column_availability(self):
 
             # Do it for index that map to columns
             if "index" in component_def:
-
                 index_def = component_def["index"]
                 if "add_labels" in index_def:
                     self._check_single_column_availability(
@@ -1290,19 +1321,12 @@ def _check_column_availability(self):
                     )
 
     def _deep_check(self):
-
         """
         Perform a "deep" check of the AnnData object using the schema definition. Adds errors to self.errors if any
 
         :rtype None
         """
 
-        # Checks that adata is fully loaded
-        if self.adata.isbacked:
-            raise RuntimeError(
-                "adata is loaded in backed mode, validation requires anndata to be loaded in memory"
-            )
-
         # Checks for deprecated columns
         self._check_deprecated_columns()
 
@@ -1352,7 +1376,6 @@ def _deep_check(self):
             )
 
     def validate_adata(self, h5ad_path: Union[str, bytes, os.PathLike] = None) -> bool:
-
         """
         Validates adata
 


Test patch:
diff --git a/cellxgene_schema_cli/tests/fixtures/examples_validate.py b/cellxgene_schema_cli/tests/fixtures/examples_validate.py
--- a/cellxgene_schema_cli/tests/fixtures/examples_validate.py
+++ b/cellxgene_schema_cli/tests/fixtures/examples_validate.py
@@ -165,7 +165,7 @@
 # ---
 # 4. Creating expression matrix,
 # X has integer values and non_raw_X has real values
-X = numpy.zeros([good_obs.shape[0], good_var.shape[0]])
+X = numpy.zeros([good_obs.shape[0], good_var.shape[0]], dtype=numpy.float32)
 non_raw_X = sparse.csr_matrix(X.copy())
 non_raw_X[0, 0] = 1.5
 
diff --git a/cellxgene_schema_cli/tests/test_schema_compliance.py b/cellxgene_schema_cli/tests/test_schema_compliance.py
--- a/cellxgene_schema_cli/tests/test_schema_compliance.py
+++ b/cellxgene_schema_cli/tests/test_schema_compliance.py
@@ -251,7 +251,9 @@ def test_obsolete_term_id(self):
         """
 
         # Not a valid term
-        self.validator.adata.obs["assay_ontology_term_id"][0] = "EFO:0009310"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "assay_ontology_term_id"
+        ] = "EFO:0009310"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -268,7 +270,10 @@ def test_assay_ontology_term_id(self):
         """
 
         # Not a valid term
-        self.validator.adata.obs["assay_ontology_term_id"][0] = "CL:000001"
+        # self.validator.adata.obs["assay_ontology_term_id"][0] = "CL:000001"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "assay_ontology_term_id"
+        ] = "CL:000001"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -279,7 +284,9 @@ def test_assay_ontology_term_id(self):
         )
 
         # Not a valid child
-        self.validator.adata.obs["assay_ontology_term_id"][0] = "EFO:0000001"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "assay_ontology_term_id"
+        ] = "EFO:0000001"
         self.validator.errors = []
         self.validator.validate_adata()
         self.assertEqual(
@@ -291,7 +298,9 @@ def test_assay_ontology_term_id(self):
         )
 
         # Includes extraneous text
-        self.validator.adata.obs["assay_ontology_term_id"][0] = "EFO:0010183 (sci-plex)"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "assay_ontology_term_id"
+        ] = "EFO:0010183 (sci-plex)"
         self.validator.errors = []
         self.validator.validate_adata()
         self.assertEqual(
@@ -308,7 +317,9 @@ def test_cell_type_ontology_term_id(self):
         """
 
         # Not a valid term
-        self.validator.adata.obs["cell_type_ontology_term_id"][0] = "EFO:0000001"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "cell_type_ontology_term_id"
+        ] = "EFO:0000001"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -326,9 +337,11 @@ def test_development_stage_ontology_term_id_human(self):
         this MUST be the most accurate HsapDv term.
         """
 
-        self.validator.adata.obs["organism_ontology_term_id"][0] = "NCBITaxon:9606"
-        self.validator.adata.obs["development_stage_ontology_term_id"][
-            0
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "organism_ontology_term_id"
+        ] = "NCBITaxon:9606"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "development_stage_ontology_term_id"
         ] = "EFO:0000001"
         self.validator.validate_adata()
         self.assertEqual(
@@ -346,11 +359,16 @@ def test_development_stage_ontology_term_id_mouse(self):
         this MUST be the most accurate MmusDv term
         """
 
-        self.validator.adata.obs["organism_ontology_term_id"][0] = "NCBITaxon:10090"
-        self.validator.adata.obs["development_stage_ontology_term_id"][
-            0
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "organism_ontology_term_id"
+        ] = "NCBITaxon:10090"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "development_stage_ontology_term_id"
         ] = "EFO:0000001"
-        self.validator.adata.obs["self_reported_ethnicity_ontology_term_id"][0] = "na"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0],
+            "self_reported_ethnicity_ontology_term_id",
+        ] = "na"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -368,11 +386,16 @@ def test_development_stage_ontology_term_id_all_species(self):
         """
 
         # Fail case not an UBERON term
-        self.validator.adata.obs["organism_ontology_term_id"][0] = "NCBITaxon:10114"
-        self.validator.adata.obs["development_stage_ontology_term_id"][
-            0
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "organism_ontology_term_id"
+        ] = "NCBITaxon:10114"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "development_stage_ontology_term_id"
         ] = "EFO:0000001"
-        self.validator.adata.obs["self_reported_ethnicity_ontology_term_id"][0] = "na"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0],
+            "self_reported_ethnicity_ontology_term_id",
+        ] = "na"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -387,11 +410,16 @@ def test_development_stage_ontology_term_id_all_species(self):
         # All other it MUST be children of UBERON:0000105 and not UBERON:0000071
         # Fail case UBERON:0000071
         self.validator.errors = []
-        self.validator.adata.obs["organism_ontology_term_id"][0] = "NCBITaxon:10114"
-        self.validator.adata.obs["development_stage_ontology_term_id"][
-            0
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "organism_ontology_term_id"
+        ] = "NCBITaxon:10114"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "development_stage_ontology_term_id"
         ] = "UBERON:0000071"
-        self.validator.adata.obs["self_reported_ethnicity_ontology_term_id"][0] = "na"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0],
+            "self_reported_ethnicity_ontology_term_id",
+        ] = "na"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -411,7 +439,9 @@ def test_disease_ontology_term_id(self):
         """
 
         # Invalid ontology
-        self.validator.adata.obs["disease_ontology_term_id"][0] = "EFO:0000001"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "disease_ontology_term_id"
+        ] = "EFO:0000001"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -423,7 +453,9 @@ def test_disease_ontology_term_id(self):
 
         # Invalid PATO term id
         self.validator.errors = []
-        self.validator.adata.obs["disease_ontology_term_id"][0] = "PATO:0001894"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "disease_ontology_term_id"
+        ] = "PATO:0001894"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -444,9 +476,12 @@ def test_self_reported_ethnicity_ontology_term_id(self):
 
         # If organism_ontolology_term_id is "NCBITaxon:9606" for Homo sapiens,
         # this MUST be either a HANCESTRO term, "multiethnic", or "unknown" if unavailable.
-        self.validator.adata.obs["organism_ontology_term_id"][0] = "NCBITaxon:9606"
-        self.validator.adata.obs["self_reported_ethnicity_ontology_term_id"][
-            0
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "organism_ontology_term_id"
+        ] = "NCBITaxon:9606"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0],
+            "self_reported_ethnicity_ontology_term_id",
         ] = "EFO:0000001"
         self.validator.validate_adata()
         self.assertEqual(
@@ -463,12 +498,15 @@ def test_self_reported_ethnicity_ontology_term_id(self):
         # development_stage_ontology_term_id has to be set to an appropriate mouse term id, otherwise there
         # will be an error in that field.
         self.validator.errors = []
-        self.validator.adata.obs["organism_ontology_term_id"][0] = "NCBITaxon:10090"
-        self.validator.adata.obs["development_stage_ontology_term_id"][
-            0
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "organism_ontology_term_id"
+        ] = "NCBITaxon:10090"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "development_stage_ontology_term_id"
         ] = "MmusDv:0000003"
-        self.validator.adata.obs["self_reported_ethnicity_ontology_term_id"][
-            0
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0],
+            "self_reported_ethnicity_ontology_term_id",
         ] = "EFO:0000001"
         self.validator.validate_adata()
         self.assertEqual(
@@ -489,9 +527,16 @@ def test_organism_ontology_term_id(self):
         # Setting "organism_ontology_term_id" to "EFO:0000001" is the fail case. However since this represents neither
         # human nor mouse, then two other columns that are dependent on it need to be set appropriately to avoid
         # other error messages: "development_stage_ontology_term_id" and "self_reported_ethnicity_ontology_term_id"
-        self.validator.adata.obs["organism_ontology_term_id"][0] = "EFO:0000001"
-        self.validator.adata.obs["development_stage_ontology_term_id"][0] = "unknown"
-        self.validator.adata.obs["self_reported_ethnicity_ontology_term_id"][0] = "na"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "organism_ontology_term_id"
+        ] = "EFO:0000001"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "development_stage_ontology_term_id"
+        ] = "unknown"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0],
+            "self_reported_ethnicity_ontology_term_id",
+        ] = "na"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -508,7 +553,9 @@ def test_tissue_ontology_term_id_base(self):
         that this cell was derived from, depending on the type of biological sample:
         """
 
-        self.validator.adata.obs["tissue_ontology_term_id"][0] = "EFO:0000001"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "tissue_ontology_term_id"
+        ] = "EFO:0000001"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -524,8 +571,8 @@ def test_tissue_ontology_term_id_cell_culture(self):
         Cell Culture - MUST be a CL term appended with " (cell culture)"
         """
 
-        self.validator.adata.obs["tissue_ontology_term_id"][
-            0
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "tissue_ontology_term_id"
         ] = "CL:0000057 (CELL culture)"
         self.validator.validate_adata()
         self.assertEqual(
@@ -542,7 +589,9 @@ def test_tissue_ontology_term_id_organoid(self):
         Organoid - MUST be an UBERON term appended with " (organoid)"
         """
 
-        self.validator.adata.obs["tissue_ontology_term_id"][0] = "CL:0000057 (ORGANOID)"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "tissue_ontology_term_id"
+        ] = "CL:0000057 (ORGANOID)"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -559,7 +608,9 @@ def test_sex_ontology_term_id(self):
         This MUST be a child of PATOPATO:0001894 for phenotypic sex or "unknown" if unavailable
         """
 
-        self.validator.adata.obs["sex_ontology_term_id"][0] = "EFO:0000001"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "sex_ontology_term_id"
+        ] = "EFO:0000001"
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
@@ -660,8 +711,12 @@ def test_suspension_type(self):
                 self.validator.warnings = []
 
                 invalid_suspension_type = "na"
-                self.validator.adata.obs["suspension_type"][1] = invalid_suspension_type
-                self.validator.adata.obs["assay_ontology_term_id"][1] = assay
+                self.validator.adata.obs.loc[
+                    self.validator.adata.obs.index[1], "suspension_type"
+                ] = invalid_suspension_type
+                self.validator.adata.obs.loc[
+                    self.validator.adata.obs.index[1], "assay_ontology_term_id"
+                ] = assay
                 self.validator.validate_adata()
                 self.assertEqual(
                     self.validator.errors,
@@ -702,8 +757,12 @@ def test_suspension_type_ancestors_inclusive(self):
                     ] = self.validator.adata.obs[
                         "suspension_type"
                     ].cat.remove_unused_categories()
-                self.validator.adata.obs["assay_ontology_term_id"][1] = assay
-                self.validator.adata.obs["suspension_type"][1] = invalid_suspension_type
+                self.validator.adata.obs.loc[
+                    self.validator.adata.obs.index[1], "assay_ontology_term_id"
+                ] = assay
+                self.validator.adata.obs.loc[
+                    self.validator.adata.obs.index[1], "suspension_type"
+                ] = invalid_suspension_type
                 self.validator.validate_adata()
                 self.assertEqual(
                     self.validator.errors,
@@ -720,10 +779,12 @@ def test_suspension_type_with_child_term_id(self):
         values depend on the assay_ontology_term_id. MUST support matching against ancestor term rules if specified.
         """
         with self.subTest("failure"):
-            self.validator.adata.obs["assay_ontology_term_id"][
-                0
+            self.validator.adata.obs.loc[
+                self.validator.adata.obs.index[0], "assay_ontology_term_id"
             ] = "EFO:0030008"  # child of EFO:0009294
-            self.validator.adata.obs["suspension_type"][0] = "nucleus"
+            self.validator.adata.obs.loc[
+                self.validator.adata.obs.index[0], "suspension_type"
+            ] = "nucleus"
 
             self.validator.validate_adata()
             self.assertEqual(
@@ -736,8 +797,8 @@ def test_suspension_type_with_child_term_id(self):
             )
 
         with self.subTest("success"):
-            self.validator.adata.obs["assay_ontology_term_id"][
-                0
+            self.validator.adata.obs.loc[
+                self.validator.adata.obs.index[0], "assay_ontology_term_id"
             ] = "EFO:0008904"  # child of EFO:0007045
             self.validator.adata.obs["suspension_type"][0] = "nucleus"
 
@@ -752,7 +813,9 @@ def test_suspension_type_unrecognized_assay(self):
         suspension_id categorical with str categories. This field MUST be "cell", "nucleus", or "na". The allowed
         values depend on the assay_ontology_term_id. MUST warn if the corresponding assay is not recognized.
         """
-        self.validator.adata.obs["assay_ontology_term_id"][1] = "EFO:0010183"
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[1], "assay_ontology_term_id"
+        ] = "EFO:0010183"
         self.validator.validate_adata()
         with self.subTest("no errors"):
             self.assertEqual(self.validator.errors, [])
@@ -813,7 +876,9 @@ def test_nan_values_must_be_rejected(self):
         """
         NaN values should not be allowed in dataframes
         """
-        self.validator.adata.obs["tissue_ontology_term_id"][0] = numpy.nan
+        self.validator.adata.obs.loc[
+            self.validator.adata.obs.index[0], "tissue_ontology_term_id"
+        ] = numpy.nan
         self.validator.validate_adata()
         self.assertEqual(
             self.validator.errors,
diff --git a/cellxgene_schema_cli/tests/test_validate.py b/cellxgene_schema_cli/tests/test_validate.py
--- a/cellxgene_schema_cli/tests/test_validate.py
+++ b/cellxgene_schema_cli/tests/test_validate.py
@@ -250,7 +250,7 @@ def validation_helper(self, matrix):
     def test_determine_seurat_convertibility(self):
         # Sparse matrix with too many nonzero values is not Seurat-convertible
         sparse_matrix_too_large = sparse.csr_matrix(
-            np.ones((good_obs.shape[0], good_var.shape[0]))
+            np.ones((good_obs.shape[0], good_var.shape[0]), dtype=np.float32)
         )
         self.validation_helper(sparse_matrix_too_large)
         self.validator._validate_seurat_convertibility()
@@ -259,7 +259,7 @@ def test_determine_seurat_convertibility(self):
 
         # Reducing nonzero count by 1, to within limit, makes it Seurat-convertible
         sparse_matrix_with_zero = sparse.csr_matrix(
-            np.ones((good_obs.shape[0], good_var.shape[0]))
+            np.ones((good_obs.shape[0], good_var.shape[0]), dtype=np.float32)
         )
         sparse_matrix_with_zero[0, 0] = 0
         self.validation_helper(sparse_matrix_with_zero)
@@ -268,7 +268,9 @@ def test_determine_seurat_convertibility(self):
         self.assertTrue(self.validator.is_seurat_convertible)
 
         # Dense matrices with a dimension that exceeds limit will fail -- zeros are irrelevant
-        dense_matrix_with_zero = np.zeros((good_obs.shape[0], good_var.shape[0]))
+        dense_matrix_with_zero = np.zeros(
+            (good_obs.shape[0], good_var.shape[0]), dtype=np.float32
+        )
         self.validation_helper(dense_matrix_with_zero)
         self.validator.schema_def["max_size_for_seurat"] = 2**2 - 1
         self.validator._validate_seurat_convertibility()
@@ -276,7 +278,7 @@ def test_determine_seurat_convertibility(self):
         self.assertFalse(self.validator.is_seurat_convertible)
 
         # Dense matrices with dimensions in bounds but total count over will succeed
-        dense_matrix = np.ones((good_obs.shape[0], good_var.shape[0]))
+        dense_matrix = np.ones((good_obs.shape[0], good_var.shape[0]), dtype=np.float32)
         self.validation_helper(dense_matrix)
         self.validator.schema_def["max_size_for_seurat"] = 2**3 - 1
         self.validator._validate_seurat_convertibility()