Repository: tompollard/tableone

Issue:
NaN SMD
Hello, thank you for the useful package.

There seems to be an issue when calculating the SMD for categorical features whereby one of the categories does not have any samples and so the entire feature is set to NaN.

```py
from tableone import TableOne
import pandas as pd
import numpy as np

df = pd.DataFrame({
    "ID": range(1, 101),
    "group": [0] * 50 + [1] * 50,  # 50 controls (0), 50 cases (1)
    "category_var": (
        ["A"] * 20 + ["B"] * 15 + ["C"] * 15 +  # Group 0 has all 3 categories
        ["A"] * 30 + ["B"] * 20                 # Group 1 is missing category "C"
    )
})

print(df.groupby(["group", "category_var"]).size().unstack())

table1 = TableOne(df, groupby="group", smd=True)
print(table1)
```

Is there a recommended way around this? I have tried modifying the code myself in the _cat_smd function to incorporate a form of the continuity correction which seems to work. I have a binary group so unsure of its effects in a multi-group comparison.

Discussion:
@tompollard: Thanks Luke, and sorry! There are a couple of open issues around handling of nans. I've been swamped with other things recently, but I'll try to get onto them over the next couple of weeks.@LukeHankey: Thanks Tom. Will check back periodically and see if I can suggest a more concrete solution.@LukeHankey: Hi @tompollard,

I may have found a solution, that at least works in my case for calculating the SMD. It is during the creating of the categorical table in `create_cat_describe`. I noticed that if there are no observations in one group for any category, the number of rows after calculating the `freq` in L249 is different for each group. 

https://github.com/tompollard/tableone/blob/31ed529fb582a9b5b2e060e591169593e967efe6/tableone/tables.py#L217-L251

I added this before L219 in the above snippet which provides all of the unique indexes from the categorical data and converting them to a string as that is what is done in L241-L246 so that indexes can match.
```py
        all_index = (
            cat_slice.melt()[['variable', 'value']]
            .dropna()
            .drop_duplicates()
            .astype(str)
        )
        all_index_str = pd.MultiIndex.from_frame(all_index)
```

After L250, I then add this so that all indexes are aligned for each group and all categories are included. All NaNs are then set to 0 as there are no observations within either group for a particular category.
```py
        df = df.reindex(all_index_str, fill_value=0)
```

Now when calculating the SMD, the proportions of each category won't be NaN. @tompollard: This issue should be addressed by https://github.com/tompollard/tableone/pull/187. Please bump to v0.9.2: https://pypi.org/project/tableone/

Solution patch:
diff --git a/tableone/tables.py b/tableone/tables.py
index 0f461b8..e4d9a61 100644
--- a/tableone/tables.py
+++ b/tableone/tables.py
@@ -216,6 +216,18 @@ class Tables:
 
         cat_slice = data[categorical].copy()
 
+        # Ensure all category levels from the full dataset are preserved
+        all_levels = {
+            col: sorted(set(cat_slice[col].dropna().astype(str)))
+            for col in categorical
+        }
+
+        # Build full multi-index with all combinations
+        full_index = pd.MultiIndex.from_tuples(
+            [(col, val) for col in categorical for val in all_levels[col]],
+            names=['variable', 'value']
+        )
+
         for g in groupbylvls:  # type: ignore
             if groupby:
                 df = cat_slice.loc[data[groupby] == g, categorical]
@@ -246,17 +258,28 @@ class Tables:
                                      in cat_slice[column].values]
 
             # create a dataframe with freq, proportion
-            df = df.melt().groupby(['variable',
-                                    'value']).size().to_frame(name='freq')
+            df = (
+                df.melt()
+                .groupby(['variable', 'value'])
+                .size()
+                .reindex(full_index, fill_value=0)
+                .to_frame(name='freq')
+            )
 
             df['percent'] = df['freq'].div(df.groupby(level=0).freq.sum(),
                                            level=0).astype(float) * 100
 
             # add row percent
-            df['percent_row'] = df['freq'].div(cat_slice[categorical]
-                                               .melt()
-                                               .groupby(['variable', 'value'])
-                                               .size()) * 100
+            full_counts = (
+                cat_slice[categorical]
+                .melt()
+                .groupby(['variable', 'value'])
+                .size()
+            )
+
+            df['percent_row'] = df.index.map(
+                lambda idx: df.at[idx, 'freq'] / full_counts.get(idx, np.nan) * 100
+            )
 
             # set number of decimal places for percent
             if isinstance(decimals, int):


Test patch:
diff --git a/tests/unit/test_tableone.py b/tests/unit/test_tableone.py
index 42902a4..10d70d0 100644
--- a/tests/unit/test_tableone.py
+++ b/tests/unit/test_tableone.py
@@ -1260,3 +1260,58 @@ class TestTableOne(object):
 
         expected = '1 (20.0)'
         assert t.tableone.loc["sex, n (%)", "None"]["Overall"] == expected
+
+
+def test_smd_with_missing_category_level_in_group():
+    """
+    Test that SMD is not NaN when one group is missing a level of a categorical variable.
+    This corresponds to GitHub issue #182.
+    """
+    df = pd.DataFrame({
+        'group': [0]*5 + [1]*5,
+        'color': ['red', 'red', 'blue', 'blue', 'green',
+                  'red', 'red', 'blue', 'blue', 'blue']
+    })
+
+    # Force color to be categorical with all expected levels
+    df['color'] = pd.Categorical(df['color'], categories=['red', 'blue', 'green'])
+
+    table = TableOne(df, columns=['color'], categorical=['color'], groupby='group', smd=True)
+
+    smd = table.tableone.loc['color, n (%)', 'Grouped by group']['SMD (0,1)'].iloc[0]
+
+    assert isinstance(smd, str) and smd.lower() != 'nan', "SMD should be computed and not be NaN"
+
+
+def test_missing_column_appears_in_first_row():
+    """
+    Test that the 'Missing' column is correctly populated only in the first
+    row of each categorical variable.
+    """
+    df = pd.DataFrame({
+        'group': [0]*5 + [1]*5,
+        'color': ['red', 'red', 'blue', 'blue', None,
+                  'red', 'red', 'blue', 'blue', 'blue']
+    })
+
+    table = TableOne(df,
+                     columns=['color'],
+                     categorical=['color'],
+                     groupby='group',
+                     include_null=False)
+
+    t1 = table.tableone
+
+    assert 'Missing' in t1['Grouped by group'].columns
+
+    # Extract rows for "color" variable
+    color_rows = t1.loc['color, n (%)', :]
+
+    # First row should contain a valid Missing value
+    assert color_rows.at[color_rows.index[0],
+                         ('Grouped by group', 'Missing')] == 1
+
+    # Remaining rows should have NaN
+    for i in range(1, len(color_rows)):
+        assert color_rows.at[color_rows.index[i],
+                             ('Grouped by group', 'Missing')] == ''