Repository: modin-project/modin

Issue:
Series.diff() internal error ends up in integrity error
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 18.04
- **Modin version** (`modin.__version__`): 0.7.3
- **Python version**: 3.6.9
- **Code we can use to reproduce**:

```python
df = ...
t = pandas.to_datetime(df['timestamp'])
delta_df = t.diff() 
delta_df = delta_df.dt.total_seconds()
```

### Describe the problem
I am using timeseries data inside the dataframe (df). Getting the Series of Datetime objects (from the index of the Dataframe) to compute the difference in time between each timeseries instance, I get an internal error in the `delta_df` object after computing the difference (first logs below).

If then I try to run the next line (delta_df.dt.total_seconds()), it throws an integrity error (second logs below).


### Source code / logs

Internal values of delta_df (from debugging):

```
'Traceback (most recent call last):\n  
File "/home/adrian.arroyo/.vscode-server/extensions/ms-python.python-2020.5.80290/pythonFiles/lib/python/debugpy/no_wheels/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_resolver.py", line 193, in _get_py_dictionary\n
    attr = getattr(var, name)\n  
File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/pandas/base.py", line 3381, in __getattribute__\n
    method = object.__getattribute__(self, item)\n  
File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/pandas/series.py", line 349, in values\n
    return super(Series, self).to_numpy().flatten()\n  
File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/pandas/base.py", line 2981, in to_numpy\n
    arr = self._query_compiler.to_numpy()\n  
File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/backends/pandas/query_compiler.py", line 182, in to_numpy\n
    len(arr) != len(self.index) or len(arr[0]) != len(self.columns)\n  
File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/error_message.py", line 53, in catch_bugs_and_request_email\n
    " caused this error.\\n{}".format(extra_log)\n
Exception: Internal Error. Please email bug_reports@modin.org with the traceback and command that caused this error.\n\n'
```

Throwed error when trying to run delta_df.dt.total_seconds():

```
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 1182, in run
    self.function(*self.args, **self.kwargs)
  File "/home/adrian.arroyo/proyectos/trainer/src/trainer.py", line 710, in easier_train
    df = trainer.set_up_timeseries_data(df, dataset_features=dataset_features, inference_features=inference_features)
  File "/home/adrian.arroyo/proyectos/trainer/src/trainer.py", line 310, in set_up_timeseries_data
    delta_df = delta_df.dt.total_seconds()
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/pandas/base.py", line 3381, in __getattribute__
    method = object.__getattribute__(self, item)
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/pandas/series.py", line 1224, in dt
    return self._default_to_pandas(pandas.Series.dt)
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/pandas/base.py", line 240, in _default_to_pandas
    pandas_obj = self._to_pandas()
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/pandas/series.py", line 1302, in _to_pandas
    df = self._query_compiler.to_pandas()
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/backends/pandas/query_compiler.py", line 89, in to_pandas
    return self._modin_frame.to_pandas()
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/engines/base/frame/data.py", line 1236, in to_pandas
    df = self._frame_mgr_cls.to_pandas(self._partitions)
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/engines/base/frame/partition_manager.py", line 329, in to_pandas
    for row in retrieved_objects
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/modin/engines/base/frame/partition_manager.py", line 330, in <listcomp>
    if not all(part.empty for part in row)
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 284, in concat
    return op.get_result()
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 497, in get_result
    mgrs_indexers, self.new_axes, concat_axis=self.axis, copy=self.copy
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 2027, in concatenate_block_managers
    return BlockManager(blocks, axes)
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 139, in __init__
    self._verify_integrity()
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 334, in _verify_integrity
    construction_error(tot_items, block.shape[1:], self.axes)
  File "/home/adrian.arroyo/.local/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 1694, in construction_error
    raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")
ValueError: Shape of passed values is (8, 1), indices imply (63, 1)
```


Discussion:
------ UPDATE ------

Just found out that it works if the DataFrame has the timestamp column set as the index.

In any case, in pandas it works without setting any index to the DataFrame  
@adr-arroyo Thanks for the report!

I believe this is an issue with `to_datetime`. We have a new implementation for this in #1444. This was merged today, are you able to see if this resolves your issue? You will need to install from master ([link to how to do this](https://modin.readthedocs.io/en/latest/installation.html#installing-from-the-github-master-branch)). Please let us know!
Hi @devin-petersohn you're welcome.

I've just updated the modin version (I also needed to install ray with pip), but now I get a different error:

```
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 1182, in run
    self.function(*self.args, **self.kwargs)
  File "/home/adrian.arroyo/proyectos/trainer/src/trainer.py", line 747, in easier_train
    df = trainer.set_up_timeseries_data(df, dataset_features=dataset_features, inference_features=inference_features)
  File "/home/adrian.arroyo/proyectos/trainer/src/trainer.py", line 343, in set_up_timeseries_data
    delta_df = delta_df.dt.total_seconds()
AttributeError: 'DatetimeProperties' object has no attribute 'total_seconds'
```

It seems the library has not the same internal functionality, do I need to change something in addition to the version?

P.D.: The variable t has internally the same issues as before. 
@adr-arroyo Thank you for the update, this is very helpful. It has uncovered some missing operations from the PR #1450.

@prutskov I notice some missing operations from your pull request:

```python
import pandas
import modin.pandas as pd

set(dir(pandas.Series.dt)) - set(dir(pd.Series.dt))
```

Output:

```
{'asfreq',
 'ceil',
 'components',
 'date',
 'day',
 'day_name',
 'dayofweek',
 'dayofyear',
 'days',
 'days_in_month',
 'daysinmonth',
 'end_time',
 'floor',
 'freq',
 'hour',
 'is_leap_year',
 'is_month_end',
 'is_month_start',
 'is_quarter_end',
 'is_quarter_start',
 'is_year_end',
 'is_year_start',
 'microsecond',
 'microseconds',
 'minute',
 'month',
 'month_name',
 'nanosecond',
 'nanoseconds',
 'normalize',
 'quarter',
 'qyear',
 'round',
 'second',
 'seconds',
 'start_time',
 'strftime',
 'time',
 'timetz',
 'to_period',
 'to_pydatetime',
 'to_pytimedelta',
 'to_timestamp',
 'total_seconds',
 'tz',
 'tz_convert',
 'tz_localize',
 'week',
 'weekday',
 'weekofyear',
 'year'}
```

Can you take a look @prutskov?
@devin-petersohn Yes, I take a look for this.

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -82,6 +82,8 @@ def dt_op_builder(df, *args, **kwargs):
         prop_val = getattr(df.squeeze().dt, property_name)
         if isinstance(prop_val, pandas.Series):
             return prop_val.to_frame()
+        elif isinstance(prop_val, pandas.DataFrame):
+            return prop_val
         else:
             return pandas.DataFrame([prop_val])
 
@@ -167,9 +169,14 @@ def default_to_pandas(self, pandas_op, *args, **kwargs):
             k: v.to_pandas if isinstance(v, type(self)) else v
             for k, v in kwargs.items()
         }
-        return self.from_pandas(
-            pandas_op(self.to_pandas(), *args, **kwargs), type(self._modin_frame)
-        )
+
+        result = pandas_op(self.to_pandas(), *args, **kwargs)
+        if isinstance(result, pandas.Series):
+            result = result.to_frame()
+        if isinstance(result, pandas.DataFrame):
+            return self.from_pandas(result, type(self._modin_frame))
+        else:
+            return result
 
     def to_pandas(self):
         return self._modin_frame.to_pandas()
@@ -592,6 +599,17 @@ def unique(self):
     dt_ceil = MapFunction.register(_dt_func_map("ceil"))
     dt_month_name = MapFunction.register(_dt_func_map("month_name"))
     dt_day_name = MapFunction.register(_dt_func_map("day_name"))
+    dt_to_pytimedelta = MapFunction.register(_dt_func_map("to_pytimedelta"))
+    dt_total_seconds = MapFunction.register(_dt_func_map("total_seconds"))
+    dt_seconds = MapFunction.register(_dt_prop_map("seconds"))
+    dt_days = MapFunction.register(_dt_prop_map("days"))
+    dt_microseconds = MapFunction.register(_dt_prop_map("microseconds"))
+    dt_nanoseconds = MapFunction.register(_dt_prop_map("nanoseconds"))
+    dt_components = MapFunction.register(_dt_prop_map("components"))
+    dt_qyear = MapFunction.register(_dt_prop_map("qyear"))
+    dt_start_time = MapFunction.register(_dt_prop_map("start_time"))
+    dt_end_time = MapFunction.register(_dt_prop_map("end_time"))
+    dt_to_timestamp = MapFunction.register(_dt_func_map("to_timestamp"))
 
     # END Dt map partitions operations
 
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -1557,6 +1557,55 @@ def month_name(self, *args, **kwargs):
     def day_name(self, *args, **kwargs):
         return Series(query_compiler=self._query_compiler.dt_day_name(*args, **kwargs))
 
+    def total_seconds(self, *args, **kwargs):
+        return Series(
+            query_compiler=self._query_compiler.dt_total_seconds(*args, **kwargs)
+        )
+
+    def to_pytimedelta(self):
+        return self._query_compiler.default_to_pandas(
+            lambda df: pandas.Series.dt.to_pytimedelta(df.squeeze().dt)
+        )
+
+    @property
+    def seconds(self):
+        return Series(query_compiler=self._query_compiler.dt_seconds())
+
+    @property
+    def days(self):
+        return Series(query_compiler=self._query_compiler.dt_days())
+
+    @property
+    def microseconds(self):
+        return Series(query_compiler=self._query_compiler.dt_microseconds())
+
+    @property
+    def nanoseconds(self):
+        return Series(query_compiler=self._query_compiler.dt_nanoseconds())
+
+    @property
+    def components(self):
+        from .dataframe import DataFrame
+
+        return DataFrame(query_compiler=self._query_compiler.dt_components())
+
+    @property
+    def qyear(self):
+        return Series(query_compiler=self._query_compiler.dt_qyear())
+
+    @property
+    def start_time(self):
+        return Series(query_compiler=self._query_compiler.dt_start_time())
+
+    @property
+    def end_time(self):
+        return Series(query_compiler=self._query_compiler.dt_end_time())
+
+    def to_timestamp(self, *args, **kwargs):
+        return Series(
+            query_compiler=self._query_compiler.dt_to_timestamp(*args, **kwargs)
+        )
+
 
 class StringMethods(object):
     def __init__(self, series):


Test patch:
diff --git a/modin/pandas/test/test_series.py b/modin/pandas/test/test_series.py
--- a/modin/pandas/test/test_series.py
+++ b/modin/pandas/test/test_series.py
@@ -1394,6 +1394,28 @@ def test_dt():
     df_equals(modin_series.dt.month_name(), pandas_series.dt.month_name())
     df_equals(modin_series.dt.day_name(), pandas_series.dt.day_name())
 
+    modin_series = pd.Series(pd.to_timedelta(np.arange(5), unit="d"))
+    pandas_series = pandas.Series(pandas.to_timedelta(np.arange(5), unit="d"))
+
+    assert_array_equal(
+        modin_series.dt.to_pytimedelta(), pandas_series.dt.to_pytimedelta()
+    )
+    df_equals(modin_series.dt.total_seconds(), pandas_series.dt.total_seconds())
+    df_equals(modin_series.dt.days, pandas_series.dt.days)
+    df_equals(modin_series.dt.seconds, pandas_series.dt.seconds)
+    df_equals(modin_series.dt.microseconds, pandas_series.dt.microseconds)
+    df_equals(modin_series.dt.nanoseconds, pandas_series.dt.nanoseconds)
+    df_equals(modin_series.dt.components, pandas_series.dt.components)
+
+    data_per = pd.date_range("1/1/2012", periods=5, freq="M")
+    pandas_series = pandas.Series(data_per, index=data_per).dt.to_period()
+    modin_series = pd.Series(data_per, index=data_per).dt.to_period()
+
+    df_equals(modin_series.dt.qyear, pandas_series.dt.qyear)
+    df_equals(modin_series.dt.start_time, pandas_series.dt.start_time)
+    df_equals(modin_series.dt.end_time, pandas_series.dt.end_time)
+    df_equals(modin_series.dt.to_timestamp(), pandas_series.dt.to_timestamp())
+
 
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 @pytest.mark.parametrize(