Repository: cytomining/pycytominer

Issue:
Fix warnings 
Fix these these warnings that pop up upon testing

```
pytest
======================================================================== test session starts ========================================================================
platform darwin -- Python 3.9.6, pytest-7.1.1, pluggy-1.0.0
rootdir: /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer
collected 129 items                                                                                                                                                 

pycytominer/tests/test_aggregate.py .........                                                                                                                 [  6%]
pycytominer/tests/test_annotate.py ...                                                                                                                        [  9%]
pycytominer/tests/test_consensus.py ..                                                                                                                        [ 10%]
pycytominer/tests/test_feature_select.py .........                                                                                                            [ 17%]
pycytominer/tests/test_normalize.py ...........                                                                                                               [ 26%]
pycytominer/tests/test_cyto_utils/test_DeepProfiler_processing.py .....                                                                                       [ 30%]
pycytominer/tests/test_cyto_utils/test_annotate_custom.py ......                                                                                              [ 34%]
pycytominer/tests/test_cyto_utils/test_cells.py .................                                                                                             [ 48%]
pycytominer/tests/test_cyto_utils/test_cp_image_features.py ..                                                                                                [ 49%]
pycytominer/tests/test_cyto_utils/test_feature_blocklist.py ..                                                                                                [ 51%]
pycytominer/tests/test_cyto_utils/test_feature_drop_outlier.py ....                                                                                           [ 54%]
pycytominer/tests/test_cyto_utils/test_feature_infer.py .....                                                                                                 [ 58%]
pycytominer/tests/test_cyto_utils/test_features_count_na.py .                                                                                                 [ 58%]
pycytominer/tests/test_cyto_utils/test_features_util.py .                                                                                                     [ 59%]
pycytominer/tests/test_cyto_utils/test_load.py ....                                                                                                           [ 62%]
pycytominer/tests/test_cyto_utils/test_modz.py .......                                                                                                        [ 68%]
pycytominer/tests/test_cyto_utils/test_output.py ......                                                                                                       [ 72%]
pycytominer/tests/test_cyto_utils/test_single_cell_ingest_utils.py ...                                                                                        [ 75%]
pycytominer/tests/test_cyto_utils/test_util.py ............                                                                                                   [ 84%]
pycytominer/tests/test_cyto_utils/test_write_gct.py ....                                                                                                      [ 87%]
pycytominer/tests/test_operations/test_correlation_threshold.py ....                                                                                          [ 90%]
pycytominer/tests/test_operations/test_get_na_columns.py ...                                                                                                  [ 93%]
pycytominer/tests/test_operations/test_sparse_random_projection.py ..                                                                                         [ 94%]
pycytominer/tests/test_operations/test_transform.py ....                                                                                                      [ 97%]
pycytominer/tests/test_operations/test_variance_threshold.py ...                                                                                              [100%]

========================================================================= warnings summary ==========================================================================
pycytominer/cyto_utils/util.py:61
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/cyto_utils/util.py:61: UserWarning: Non-canonical compartment detected: new
    warnings.warn(warn_str)

pycytominer/tests/test_consensus.py: 6 warnings
pycytominer/tests/test_feature_select.py: 5 warnings
pycytominer/tests/test_cyto_utils/test_modz.py: 20 warnings
pycytominer/tests/test_cyto_utils/test_util.py: 1 warning
pycytominer/tests/test_operations/test_correlation_threshold.py: 5 warnings
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/cyto_utils/util.py:338: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    np.tril(np.ones(data_cor_natri_df.shape), k=-1).astype(np.bool)

pycytominer/tests/test_feature_select.py::test_feature_select_noise_removal
pycytominer/tests/test_feature_select.py::test_feature_select_noise_removal
pycytominer/tests/test_feature_select.py::test_feature_select_noise_removal
pycytominer/tests/test_feature_select.py::test_feature_select_noise_removal
pycytominer/tests/test_feature_select.py::test_feature_select_noise_removal
pycytominer/tests/test_feature_select.py::test_feature_select_noise_removal
pycytominer/tests/test_feature_select.py::test_feature_select_noise_removal
  /usr/local/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3613: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.
    return std(axis=axis, dtype=dtype, out=out, ddof=ddof, **kwargs)

pycytominer/tests/test_normalize.py::test_normalize_robustize_mad_allsamples
pycytominer/tests/test_normalize.py::test_normalize_robustize_mad_allsamples_novar
pycytominer/tests/test_operations/test_transform.py::test_robust_mad
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/operations/transform.py:131: DeprecationWarning: `median_absolute_deviation` is deprecated, use `median_abs_deviation` instead!
  
  To preserve the existing default behavior, use
  `scipy.stats.median_abs_deviation(..., scale=1/1.4826)`.
  The value 1.4826 is not numerically precise for scaling
  with a normal distribution. For a numerically precise value, use
  `scipy.stats.median_abs_deviation(..., scale='normal')`.
  
    median_absolute_deviation(X, nan_policy="omit"), index=self.median.index

pycytominer/tests/test_cyto_utils/test_cells.py::test_merge_single_cells
pycytominer/tests/test_cyto_utils/test_cells.py::test_merge_single_cells
pycytominer/tests/test_cyto_utils/test_cells.py::test_merge_single_cells
pycytominer/tests/test_cyto_utils/test_cells.py::test_merge_single_cells
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/cyto_utils/cells.py:693: FutureWarning: Passing 'suffixes' which cause duplicate columns {'ObjectNumber_cytoplasm'} in the result is deprecated and will raise a MergeError in a future version.
    sc_df = sc_df.merge(

pycytominer/tests/test_cyto_utils/test_modz.py::test_modz_base
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/tests/test_cyto_utils/test_modz.py:48: FutureWarning: The 'check_less_precise' keyword in testing.assert_*_equal is deprecated and will be removed in a future version. You can stop passing 'check_less_precise' to silence this warning.
    pd.testing.assert_series_equal(

pycytominer/tests/test_cyto_utils/test_modz.py::test_modz
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/tests/test_cyto_utils/test_modz.py:71: FutureWarning: The 'check_less_precise' keyword in testing.assert_*_equal is deprecated and will be removed in a future version. You can stop passing 'check_less_precise' to silence this warning.
    pd.testing.assert_frame_equal(

pycytominer/tests/test_cyto_utils/test_modz.py::test_modz_multiple_columns
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/tests/test_cyto_utils/test_modz.py:118: FutureWarning: The 'check_less_precise' keyword in testing.assert_*_equal is deprecated and will be removed in a future version. You can stop passing 'check_less_precise' to silence this warning.
    pd.testing.assert_frame_equal(

pycytominer/tests/test_cyto_utils/test_modz.py::test_modz_multiple_columns_one_metadata_column
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/tests/test_cyto_utils/test_modz.py:148: FutureWarning: The 'check_less_precise' keyword in testing.assert_*_equal is deprecated and will be removed in a future version. You can stop passing 'check_less_precise' to silence this warning.
    pd.testing.assert_frame_equal(

pycytominer/tests/test_cyto_utils/test_output.py::test_compress
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/tests/test_cyto_utils/test_output.py:57: FutureWarning: The 'check_less_precise' keyword in testing.assert_*_equal is deprecated and will be removed in a future version. You can stop passing 'check_less_precise' to silence this warning.
    pd.testing.assert_frame_equal(

pycytominer/tests/test_cyto_utils/test_output.py::test_compress_tsv
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/tests/test_cyto_utils/test_output.py:74: FutureWarning: The 'check_less_precise' keyword in testing.assert_*_equal is deprecated and will be removed in a future version. You can stop passing 'check_less_precise' to silence this warning.
    pd.testing.assert_frame_equal(

pycytominer/tests/test_cyto_utils/test_output.py::test_output_none
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/tests/test_cyto_utils/test_output.py:90: FutureWarning: The 'check_less_precise' keyword in testing.assert_*_equal is deprecated and will be removed in a future version. You can stop passing 'check_less_precise' to silence this warning.
    pd.testing.assert_frame_equal(

pycytominer/tests/test_operations/test_transform.py::test_low_variance_spherize
  /usr/local/lib/python3.9/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide
    c /= stddev[:, None]

pycytominer/tests/test_operations/test_transform.py::test_low_variance_spherize
  /usr/local/lib/python3.9/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide
    c /= stddev[None, :]

pycytominer/tests/test_operations/test_transform.py::test_robust_mad
  /Users/shsingh/work/repos/software/cytominer/cytomining/pycytominer/pycytominer/tests/test_operations/test_transform.py:86: DeprecationWarning: `median_absolute_deviation` is deprecated, use `median_abs_deviation` instead!
  
  To preserve the existing default behavior, use
  `scipy.stats.median_abs_deviation(..., scale=1/1.4826)`.
  The value 1.4826 is not numerically precise for scaling
  with a normal distribution. For a numerically precise value, use
  `scipy.stats.median_abs_deviation(..., scale='normal')`.
  
    result = median_absolute_deviation(transform_df).sum()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================ 129 passed, 62 warnings in 63.04s (0:01:03) ============================================================
(pycytominer) pycytominer$ 
```

Discussion:

Patch:
diff --git a/pycytominer/cyto_utils/util.py b/pycytominer/cyto_utils/util.py
index be22262..a7c5107 100644
--- a/pycytominer/cyto_utils/util.py
+++ b/pycytominer/cyto_utils/util.py
@@ -349,7 +349,7 @@ def get_pairwise_correlation(population_df, method="pearson"):
 
     # Replace upper triangle in correlation matrix with NaN
     data_cor_natri_df = data_cor_natri_df.where(
-        np.tril(np.ones(data_cor_natri_df.shape), k=-1).astype(np.bool)
+        np.tril(np.ones(data_cor_natri_df.shape), k=-1).astype(bool)
     )
 
     # Acquire pairwise correlations in a long format
diff --git a/pycytominer/operations/noise_removal.py b/pycytominer/operations/noise_removal.py
index 3325108..57ef4f0 100644
--- a/pycytominer/operations/noise_removal.py
+++ b/pycytominer/operations/noise_removal.py
@@ -71,7 +71,7 @@ def noise_removal(
     population_df = population_df.assign(group_id=group_info)
 
     # Get the standard deviations of features within each group
-    stdev_means_df = population_df.groupby("group_id").apply(lambda x: np.std(x)).mean()
+    stdev_means_df = population_df.groupby("group_id").std(ddof=0).mean()
 
     # Identify noisy features with a greater mean stdev within perturbation group than the threshold
     to_remove = stdev_means_df[
diff --git a/pycytominer/operations/transform.py b/pycytominer/operations/transform.py
index 5315958..59977da 100644
--- a/pycytominer/operations/transform.py
+++ b/pycytominer/operations/transform.py
@@ -9,7 +9,7 @@ import os
 import numpy as np
 import pandas as pd
 from scipy.linalg import eigh
-from scipy.stats import median_absolute_deviation
+from scipy.stats import median_abs_deviation
 from sklearn.base import BaseEstimator, TransformerMixin
 
 
@@ -169,8 +169,10 @@ class RobustMAD(BaseEstimator, TransformerMixin):
         """
         # Get the mean of the features (columns) and center if specified
         self.median = X.median()
+        # The scale param is required to preserve previous behavior. More info at:
+        # https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.median_absolute_deviation.html#scipy.stats.median_absolute_deviation
         self.mad = pd.Series(
-            median_absolute_deviation(X, nan_policy="omit"), index=self.median.index
+            median_abs_deviation(X, nan_policy="omit", scale=1/1.4826), index=self.median.index
         )
         return self
 


Test patch:
diff --git a/pycytominer/tests/test_cyto_utils/test_cells.py b/pycytominer/tests/test_cyto_utils/test_cells.py
index 1884168..2f86849 100644
--- a/pycytominer/tests/test_cyto_utils/test_cells.py
+++ b/pycytominer/tests/test_cyto_utils/test_cells.py
@@ -4,6 +4,7 @@ import pytest
 import tempfile
 import pandas as pd
 from sqlalchemy import create_engine
+import pytest
 
 from pycytominer import aggregate, normalize
 from pycytominer.cyto_utils.cells import SingleCells, _sqlite_strata_conditions
@@ -135,12 +136,15 @@ ap_subsample = SingleCells(
     subsample_n=2,
     subsampling_random_state=123,
 )
-ap_new = SingleCells(
-    sql_file=new_file,
-    load_image_data=False,
-    compartments=new_compartments,
-    compartment_linking_cols=new_linking_cols,
-)
+
+# Warning expected for compartment "new" because is not in default compartment list.
+with pytest.warns(UserWarning, match='Non-canonical compartment detected: new'):
+    ap_new = SingleCells(
+        sql_file=new_file,
+        load_image_data=False,
+        compartments=new_compartments,
+        compartment_linking_cols=new_linking_cols,
+    )
 
 ap_image_all_features = SingleCells(
     sql_file=image_file,
diff --git a/pycytominer/tests/test_cyto_utils/test_modz.py b/pycytominer/tests/test_cyto_utils/test_modz.py
index 31a0a37..8a02646 100644
--- a/pycytominer/tests/test_cyto_utils/test_modz.py
+++ b/pycytominer/tests/test_cyto_utils/test_modz.py
@@ -46,7 +46,7 @@ def test_modz_base():
     consensus_df = modz_base(data_df, min_weight=1, precision=precision)
     expected_result = data_df.mean().round(precision)
     pd.testing.assert_series_equal(
-        expected_result, consensus_df, check_less_precise=True
+        expected_result, consensus_df, check_exact=False, atol=1e-3
     )
 
 
@@ -69,7 +69,7 @@ def test_modz():
     expected_result = data_replicate_df.groupby(replicate_columns).mean().round(4)
     expected_result.index.name = replicate_columns
     pd.testing.assert_frame_equal(
-        expected_result.reset_index(), consensus_df, check_less_precise=True
+        expected_result.reset_index(), consensus_df, check_exact=False, atol=1e-3
     )
 
 
@@ -116,7 +116,7 @@ def test_modz_multiple_columns():
     expected_result = data_replicate_multi_df.groupby(replicate_columns).mean().round(4)
 
     pd.testing.assert_frame_equal(
-        expected_result.reset_index(), consensus_df, check_less_precise=True
+        expected_result.reset_index(), consensus_df, check_exact=False, atol=1e-3
     )
 
 
@@ -146,7 +146,7 @@ def test_modz_multiple_columns_one_metadata_column():
     expected_result = data_replicate_multi_df.groupby(replicate_columns).mean().round(4)
     expected_result.index.name = replicate_columns
     pd.testing.assert_frame_equal(
-        expected_result.reset_index(), consensus_df, check_less_precise=True
+        expected_result.reset_index(), consensus_df, check_exact=False, atol=1e-3
     )
 
 
diff --git a/pycytominer/tests/test_cyto_utils/test_output.py b/pycytominer/tests/test_cyto_utils/test_output.py
index c66c28d..690f051 100644
--- a/pycytominer/tests/test_cyto_utils/test_output.py
+++ b/pycytominer/tests/test_cyto_utils/test_output.py
@@ -55,7 +55,7 @@ def test_compress():
     result = pd.read_csv(output_filename)
 
     pd.testing.assert_frame_equal(
-        result, data_df, check_names=False, check_less_precise=1
+        result, data_df, check_names=False, check_exact=False, atol=1e-3
     )
 
 
@@ -72,7 +72,7 @@ def test_compress_tsv():
 
     result = pd.read_csv(output_filename, sep="\t")
     pd.testing.assert_frame_equal(
-        result, data_df, check_names=False, check_less_precise=1
+        result, data_df, check_names=False, check_exact=False, atol=1e-3
     )
 
 
@@ -88,7 +88,7 @@ def test_output_none():
 
     result = pd.read_csv(output_filename)
     pd.testing.assert_frame_equal(
-        result, data_df, check_names=False, check_less_precise=1
+        result, data_df, check_names=False, check_exact=False, atol=1e-3
     )
 
 
diff --git a/pycytominer/tests/test_operations/test_transform.py b/pycytominer/tests/test_operations/test_transform.py
index fa22d54..90566e0 100644
--- a/pycytominer/tests/test_operations/test_transform.py
+++ b/pycytominer/tests/test_operations/test_transform.py
@@ -3,7 +3,7 @@ import random
 import pytest
 import numpy as np
 import pandas as pd
-from scipy.stats import median_absolute_deviation
+from scipy.stats import median_abs_deviation
 from pycytominer.operations.transform import Spherize, RobustMAD
 
 random.seed(123)
@@ -83,7 +83,7 @@ def test_robust_mad():
     assert int(result) == expected_result
 
     # Check a median absolute deviation equal to the number of columns
-    result = median_absolute_deviation(transform_df).sum()
+    result = median_abs_deviation(transform_df, scale=1/1.4826).sum()
     expected_result = data_df.shape[1]
 
     assert int(result) == expected_result