Repository: pandas-dev/pandas

Issue:
StataReader selectively ignores format details
Code Sample, a copy-pastable example if possible

[link to current commit](https://github.com/pandas-dev/pandas/blob/22515f58c178cdb5cd38c4e56f26dc91c7053550/pandas/io/stata.py#L1032-L1034)

```python
    def _read_header(self):
        # ...
        # remove format details from %td
        self.fmtlist = ["%td" if x.startswith("%td") else x
                        for x in self.fmtlist]
```

which allows for this check to succeed when format details are present

```python
def _stata_elapsed_date_to_datetime_vec(dates, fmt):
    # ...
    elif fmt in ["%td", "td", "%d", "d"]:  # Delta days relative to base
        # ...
```

#### Problem description

It looks like `pandas` is attempting to _the right thing_ by dropping format codes from `%td` formats so that they will [match the correct format](https://github.com/pandas-dev/pandas/blob/22515f58c178cdb5cd38c4e56f26dc91c7053550/pandas/io/stata.py#L321) in `_stata_elapsed_date_to_datetime_vec`

I handle a large number of Stata files and arbitrarily supporting display formats with only `%td` is problematic.

#### Expected Output

Ideally none of these four assert statements would fail

```python
import datetime as dt
import pandas

# File attached below
d1, d2, d3, d4 = pandas.read_stata('STATA_DATES.DTA').iloc[0]

expected = dt.datetime(2006, 11, 21)

assert d1 == expected  # fails    - format is %dD_m_Y
assert d2 == expected  # succeeds - format is %tdD_m_Y
assert d3 == expected  # fails    - format is %tcD_m_Y
assert d4 == expected  # succeeds - format is %tc
````

#### Proposed Fixes

1. Strip formats from other types by adding extra list comprehensions in `_read_header`. My current workaround is to subclass `StataReader` and add this logic after `_read_header`.

1. Use `str.startswith`'s lesser-known tuple argument
   In `_stata_elapsed_date_to_datetime_vec` change lines like
  `elif fmt in ["%td", "td", "%d", "d"]:  # Delta days relative to base`
   to
   `elif fmt.startswith(("%td", "td", "%d", "d")):  # Delta days relative to base`
   this obviates the need for the list comprehensions in the first place.

#### Output of ``pd.show_versions()``

<details>
>>> pandas.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Darwin
OS-release: 16.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: None
pip: 9.0.1
setuptools: 27.2.0
Cython: None
numpy: 1.12.1
scipy: None
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: 1.0.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
boto: None
pandas_datareader: None

</details>

#### Sample Files
[STATA_DATES.DTA.zip](https://github.com/pandas-dev/pandas/files/1360608/STATA_DATES.DTA.zip) the DTA file used for the *Expected Output*. It's a single row with 4 values


![stata date types](https://user-images.githubusercontent.com/26555712/31249289-a4652626-a9cb-11e7-927c-b822edc723d0.png)

The file was generated from this CSV file with the following commands. 
[stata_dates.csv.zip](https://github.com/pandas-dev/pandas/files/1360647/stata_dates.csv.zip)
```
load ~/tmp/stata_dates.csv
format date_d %dD_m_Y
format date_td %tdD_m_Y


format datetime_format %tcD_m_Y
format datetime_noformat %tc
save ~/tmp/STATA_DATE_D_TD.DTA
```


Discussion:
cc @bashtage - I don't know anything about Stata but at a surface level this seems reasonable, I definitely welcome you to submit a PR.
Sounds reasonable -- seems like pure additional capability without any changes to existing features.  Only small concern is whether it is easy to comprehensively test this feature?
Is the proposal to do 1 or 2? or 1 and 2?  If it is -or- I think 2 is probably cleaner -- handle dates where dates are handled.
Unless requested otherwise I will pursue proposal 2.

It *does* change existing behavior - instead of returning a numeric value a datetime will be returned instead. Will that be a problem?

For comprehensive testing, what do you have in mind? I see a `pandas/tests/io/data` directory containing fixtures for (among others) stata. I'm happy to generate a file similar to the attached earlier with a larger set of fields.

Fundamentally I know of two types of Stata date/time fields - those with day precision (val += 1 adds a single day) and with microsecond precision.

Everything beyond that I understand only to be formatting options similar to `strftime`.

Also there's the wrinkle of `%tC` accounting for leap seconds and `%tc` ignoring leap seconds, but pandas already handles this
I agree that this is a minor change in behavior, but I don't think many people would really want numbers for `%tdD...` 

Resolving patch:
diff --git a/pandas/io/stata.py b/pandas/io/stata.py
--- a/pandas/io/stata.py
+++ b/pandas/io/stata.py
@@ -306,11 +306,11 @@ def convert_delta_safe(base, deltas, unit):
         data_col[bad_locs] = 1.0  # Replace with NaT
     dates = dates.astype(np.int64)
 
-    if fmt in ["%tc", "tc"]:  # Delta ms relative to base
+    if fmt.startswith(("%tc", "tc")):  # Delta ms relative to base
         base = stata_epoch
         ms = dates
         conv_dates = convert_delta_safe(base, ms, 'ms')
-    elif fmt in ["%tC", "tC"]:
+    elif fmt.startswith(("%tC", "tC")):
         from warnings import warn
 
         warn("Encountered %tC format. Leaving in Stata Internal Format.")
@@ -318,27 +318,30 @@ def convert_delta_safe(base, deltas, unit):
         if has_bad_values:
             conv_dates[bad_locs] = pd.NaT
         return conv_dates
-    elif fmt in ["%td", "td", "%d", "d"]:  # Delta days relative to base
+    # Delta days relative to base
+    elif fmt.startswith(("%td", "td", "%d", "d")):
         base = stata_epoch
         days = dates
         conv_dates = convert_delta_safe(base, days, 'd')
-    elif fmt in ["%tw", "tw"]:  # does not count leap days - 7 days is a week
+    # does not count leap days - 7 days is a week.
+    # 52nd week may have more than 7 days
+    elif fmt.startswith(("%tw", "tw")):
         year = stata_epoch.year + dates // 52
         days = (dates % 52) * 7
         conv_dates = convert_year_days_safe(year, days)
-    elif fmt in ["%tm", "tm"]:  # Delta months relative to base
+    elif fmt.startswith(("%tm", "tm")):  # Delta months relative to base
         year = stata_epoch.year + dates // 12
         month = (dates % 12) + 1
         conv_dates = convert_year_month_safe(year, month)
-    elif fmt in ["%tq", "tq"]:  # Delta quarters relative to base
+    elif fmt.startswith(("%tq", "tq")):  # Delta quarters relative to base
         year = stata_epoch.year + dates // 4
         month = (dates % 4) * 3 + 1
         conv_dates = convert_year_month_safe(year, month)
-    elif fmt in ["%th", "th"]:  # Delta half-years relative to base
+    elif fmt.startswith(("%th", "th")):  # Delta half-years relative to base
         year = stata_epoch.year + dates // 2
         month = (dates % 2) * 6 + 1
         conv_dates = convert_year_month_safe(year, month)
-    elif fmt in ["%ty", "ty"]:  # Years -- not delta
+    elif fmt.startswith(("%ty", "ty")):  # Years -- not delta
         year = dates
         month = np.ones_like(dates)
         conv_dates = convert_year_month_safe(year, month)
@@ -1029,10 +1032,6 @@ def _read_header(self):
         # calculate size of a data record
         self.col_sizes = lmap(lambda x: self._calcsize(x), self.typlist)
 
-        # remove format details from %td
-        self.fmtlist = ["%td" if x.startswith("%td") else x
-                        for x in self.fmtlist]
-
     def _read_new_header(self, first_char):
         # The first part of the header is common to 117 and 118.
         self.path_or_buf.read(27)  # stata_dta><header><release>
@@ -1578,7 +1577,8 @@ def read(self, nrows=None, convert_dates=None,
         self._do_convert_missing(data, convert_missing)
 
         if convert_dates:
-            cols = np.where(lmap(lambda x: x in _date_formats,
+            cols = np.where(lmap(lambda x: any(x.startswith(fmt)
+                                               for fmt in _date_formats),
                                  self.fmtlist))[0]
             for i in cols:
                 col = data.columns[i]


Test patch:
diff --git a/pandas/tests/io/data/stata13_dates.dta b/pandas/tests/io/data/stata13_dates.dta
new file mode 100644
Binary files /dev/null and b/pandas/tests/io/data/stata13_dates.dta differ
diff --git a/pandas/tests/io/test_stata.py b/pandas/tests/io/test_stata.py
--- a/pandas/tests/io/test_stata.py
+++ b/pandas/tests/io/test_stata.py
@@ -96,6 +96,8 @@ def setup_method(self, method):
 
         self.dta24_111 = os.path.join(self.dirpath, 'stata7_111.dta')
 
+        self.stata_dates = os.path.join(self.dirpath, 'stata13_dates.dta')
+
     def read_dta(self, file):
         # Legacy default reader configuration
         return read_stata(file, convert_dates=True)
@@ -1327,3 +1329,22 @@ def test_set_index(self):
             df.to_stata(path)
             reread = pd.read_stata(path, index_col='index')
         tm.assert_frame_equal(df, reread)
+
+    @pytest.mark.parametrize(
+        'column', ['ms', 'day', 'week', 'month', 'qtr', 'half', 'yr'])
+    def test_date_parsing_ignores_format_details(self, column):
+        # GH 17797
+        #
+        # Test that display formats are ignored when determining if a numeric
+        # column is a date value.
+        #
+        # All date types are stored as numbers and format associated with the
+        # column denotes both the type of the date and the display format.
+        #
+        # STATA supports 9 date types which each have distinct units. We test 7
+        # of the 9 types, ignoring %tC and %tb. %tC is a variant of %tc that
+        # accounts for leap seconds and %tb relies on STATAs business calendar.
+        df = read_stata(self.stata_dates)
+        unformatted = df.loc[0, column]
+        formatted = df.loc[0, column + "_fmt"]
+        assert unformatted == formatted