Repository: pandas-dev/pandas
Pull request: Lint rule to catch incorrect sphinx directives
- [x] closes #18425
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Problem statement:
Lint rule to catch incorrect sphinx directives
#### Code Sample, a copy-pastable example if possible
None. Documentation / build issue.

#### Problem description

There are few places where tag "versionadded" misses one semicolon, making it a sphinx comment instead of directive:
```
pandas/core/frame.py
4516:            .. versionadded: 0.18.0
4679:            .. versionadded: 0.16.1

pandas/core/generic.py
968:            .. versionadded: 0.21.0

pandas/core/series.py
1629:            .. versionadded: 0.19.0
2216:            .. versionadded: 0.18.0

pandas/core/tools/datetimes.py
117:        .. versionadded: 0.18.1
143:        .. versionadded: 0.16.1
181:        .. versionadded: 0.20.0
187:        .. versionadded: 0.22.0

pandas/tseries/offsets.py
778:    .. versionadded: 0.16.1
882:    .. versionadded: 0.18.1
```

Correct way to use it: `.. versionadded:: x.xx.x"` (note two semicolons).

Apart from fixing it here and now, it would be useful to have a lint rule that warns contributors about this and similar issues in future (in docstrings and in docs).

#### Expected Output

Lint rule to catch incorrect sphinx directives.

#### Output of ``pd.show_versions()``
Doesn't matter.

Hints from issue discussion:

Commit message: DOC: fix incorrect sphinx directives

Patch to review:
diff --git a/ci/lint.sh b/ci/lint.sh
index 4027737900bf9..06a7ac485bfed 100755
--- a/ci/lint.sh
+++ b/ci/lint.sh
@@ -90,13 +90,26 @@ if [ "$LINT" ]; then
     #
     # Check the following functions:
     # any(), all(), sum(), max(), min(), list(), dict(), set(), frozenset(), tuple(), str.join()
-    grep -R --include="*.py*" -E "[^_](any|all|sum|max|min|list|dict|set|frozenset|tuple|join)\(\[.* for .* in .*\]\)"
+    grep -R --include="*.py*" -E "[^_](any|all|sum|max|min|list|dict|set|frozenset|tuple|join)\(\[.* for .* in .*\]\)" *
 
     if [ $? = "0" ]; then
         RET=1
     fi
     echo "Check for use of lists instead of generators in built-in Python functions DONE"
 
+    echo "Check for incorrect sphinx directives"
+    SPHINX_DIRECTIVES=$(echo \
+       "autosummary|contents|currentmodule|deprecated|function|image|"\
+       "important|include|ipython|literalinclude|math|module|note|raw|"\
+       "seealso|toctree|versionadded|versionchanged|warning" | tr -d "[:space:]")
+    for path in './pandas' './doc/source'
+    do
+        grep -R --include="*.py" --include="*.rst" -E "\.\. ($SPHINX_DIRECTIVES):[^:]" $path
+        if [ $? = "0" ]; then
+            RET=1
+        fi
+    done
+    echo "Check for incorrect sphinx directives DONE"
 else
     echo "NOT Linting"
 fi
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index 7145fa709c345..f847fbebb091a 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -4513,7 +4513,7 @@ def unstack(self, level=-1, fill_value=None):
         fill_value : replace NaN with this value if the unstack produces
             missing values
 
-            .. versionadded: 0.18.0
+            .. versionadded:: 0.18.0
 
         See also
         --------
@@ -4676,7 +4676,7 @@ def diff(self, periods=1, axis=0):
         axis : {0 or 'index', 1 or 'columns'}, default 0
             Take difference over rows (0) or columns (1).
 
-            .. versionadded: 0.16.1
+            .. versionadded:: 0.16.1
 
         Returns
         -------
diff --git a/pandas/core/series.py b/pandas/core/series.py
index be1de4c6814ba..d7833526c0408 100644
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -1626,7 +1626,7 @@ def append(self, to_append, ignore_index=False, verify_integrity=False):
         ignore_index : boolean, default False
             If True, do not use the index labels.
 
-            .. versionadded: 0.19.0
+            .. versionadded:: 0.19.0
 
         verify_integrity : boolean, default False
             If True, raise Exception on creating index with duplicates
@@ -2213,7 +2213,7 @@ def unstack(self, level=-1, fill_value=None):
         fill_value : replace NaN with this value if the unstack produces
             missing values
 
-            .. versionadded: 0.18.0
+            .. versionadded:: 0.18.0
 
         Examples
         --------
diff --git a/pandas/core/tools/datetimes.py b/pandas/core/tools/datetimes.py
index 65f4704da3800..219fb3f67db97 100644
--- a/pandas/core/tools/datetimes.py
+++ b/pandas/core/tools/datetimes.py
@@ -114,7 +114,7 @@ def to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False,
     ----------
     arg : integer, float, string, datetime, list, tuple, 1-d array, Series
 
-        .. versionadded: 0.18.1
+        .. versionadded:: 0.18.1
 
            or DataFrame/dict-like
 
@@ -140,7 +140,7 @@ def to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False,
         Warning: yearfirst=True is not strict, but will prefer to parse
         with year first (this is a known bug, based on dateutil beahavior).
 
-        .. versionadded: 0.16.1
+        .. versionadded:: 0.16.1
 
     utc : boolean, default None
         Return UTC DatetimeIndex if True (converting any tz-aware
@@ -178,13 +178,13 @@ def to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False,
         - If Timestamp convertible, origin is set to Timestamp identified by
           origin.
 
-        .. versionadded: 0.20.0
+        .. versionadded:: 0.20.0
     cache : boolean, default False
         If True, use a cache of unique, converted dates to apply the datetime
         conversion. May produce sigificant speed-up when parsing duplicate date
         strings, especially ones with timezone offsets.
 
-        .. versionadded: 0.22.0
+        .. versionadded:: 0.22.0
 
     Returns
     -------
diff --git a/pandas/tseries/offsets.py b/pandas/tseries/offsets.py
index 2097fb22b3ec5..eef9d165e2447 100644
--- a/pandas/tseries/offsets.py
+++ b/pandas/tseries/offsets.py
@@ -774,7 +774,7 @@ class BusinessHour(BusinessHourMixin, SingleConstructorOffset):
     """
     DateOffset subclass representing possibly n business days
 
-    .. versionadded: 0.16.1
+    .. versionadded:: 0.16.1
 
     """
     _prefix = 'BH'
@@ -878,7 +878,7 @@ class CustomBusinessHour(BusinessHourMixin, SingleConstructorOffset):
     """
     DateOffset subclass representing possibly n custom business days
 
-    .. versionadded: 0.18.1
+    .. versionadded:: 0.18.1
 
     """
     _prefix = 'CBH'


Code review comments:
diff_hunk: @@ -90,13 +90,26 @@ if [ "$LINT" ]; then
     #
     # Check the following functions:
     # any(), all(), sum(), max(), min(), list(), dict(), set(), frozenset(), tuple(), str.join()
-    grep -R --include="*.py*" -E "[^_](any|all|sum|max|min|list|dict|set|frozenset|tuple|join)\(\[.* for .* in .*\]\)"
+    grep -R --include="*.py*" -E "[^_](any|all|sum|max|min|list|dict|set|frozenset|tuple|join)\(\[.* for .* in .*\]\)" *
 
     if [ $? = "0" ]; then
         RET=1
     fi
     echo "Check for use of lists instead of generators in built-in Python functions DONE"
 
+    echo "Check for incorrect sphinx directives"
+    SPHINX_DIRECTIVES=$(echo \
+       "autosummary|contents|currentmodule|deprecated|function|image|"\
+       "important|include|ipython|literalinclude|math|module|note|raw|"\
+       "seealso|toctree|versionadded|versionchanged|warning" | tr -d "[:space:]")
+    for path in './pandas' './doc/source'
+    do
+        grep -R --include="*.py" --include="*.rst" -E "\.\. ($SPHINX_DIRECTIVES):[^:]" $path
path: ci/lint.sh
text: @user1:
could include *.pyx here as well (I don't know if we actually have doc-strings that include directives in cython, but can't hurt).