ENH: Adding origin parameter in pd.to_datetime
State: closed

Fixes #11276
Fixes  #11745


Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype

Comments:
tests!

On it. Any boundary conditions that I must take into consideration?

what do you mean?

also this would need to be added in Timestamp as that accepts a unit as well 

Apologies for the late reply. I am not able to understand which file to edit for Timestamp. Kindly guide, @jreback? 

https://github.com/pydata/pandas/blob/master/pandas/tslib.pyx#L222

Thanks @jreback. Will go through the file asap :-) 

Hi, I am bit confused regarding the implementation. As far as I could understand, following might help. Kindly correct me if I am wrong.

```
## Constructor in Timestamp Class.
def __new__(cls, object ts_input, object offset=None, tz=None, unit=None):
        cdef _TSObject ts
        cdef _Timestamp ts_base

        if unit == 'julian':
            unit = 'D'
            ts_input = ts_input - Timestamp(0).to_julian_date()

        ts = convert_to_tsobject(ts_input, tz, unit)

        if ts.value == NPY_NAT:
            return NaT

        if util.is_string_object(offset):
            from pandas.tseries.frequencies import to_offset
            offset = to_offset(offset)

        # make datetime happy
        ts_base = _Timestamp.__new__(cls, ts.dts.year, ts.dts.month,
                                     ts.dts.day, ts.dts.hour, ts.dts.min,
                                     ts.dts.sec, ts.dts.us, ts.tzinfo)

        # fill out rest of data
        ts_base.value = ts.value
        ts_base.offset = offset
        ts_base.nanosecond = ts.dts.ps / 1000

        return ts_base
```

Does it seem okay, @jreback? 

@sumitbinnani this should be implemented in `cast_from_unit`

this needs tests

see #11745; let's call this `origin` (and in this case I think it would be a string, e.g. 'julian', rather than a specific date)

Have been a bit occupied for last a few weeks owing to applications and submission deadlines. Will make the necessary changes for #11745 asap. Facing a bit difficulty writing test cases (going through documentations and will mostly be done by this weekend).

Have changed the definition of  `to_datetime` to include a new parameter `origin`, and have added following lines in the code:

```
    ...
    if origin is not None:
        import datetime
        from pandas.core.api import Timestamp
        if origin == 'julian':
            arg = arg - Timestamp(0).to_julian_date()
        elif isinstance(origin, datetime.date):
            arg = arg + Timestamp(origin).to_julian_date() - Timestamp(0).to_julian_date()

   return ...
```

Should I remove the changes made to the class `Timestamp` as `unit` is no longer being used for the required functionality? @jreback?

@sumitbinnani yes, remove the original `unit` changes and go with `origin` as discussed

Hi @jreback, 

Have made the changes as followed:
- Added `origin` as a new parameter with default value of `None` to `to_datetime`
- Updated documentation for `to_datetime`
- Added test case for the patch (wrote test case for the first time and might require a bit of changes)

Let me know if any additional changes are required.

pls add whatsnew note in Enhancements for 0.18.0

Have incorporated the changes in Code Review.

The file for what's new in 0.18.0, is in commit ahead of my branch's head. The rebase command isn't working on my system (maybe configuration issue). Should I copy the file from master branch and make changes, or should I submit a new pull requests for the same?

Can you just try:

```
git fetch upstream
git rebase -i upstream/master
```

Does this not work?

pls add a whatsnew entry (in enhancements)

@jorisvandenbossche, I am getting the following error:

```
git: 'rebase' is not a git command. See 'git --help'.
```

The error started popping a week back, and I guess is due to the configuration problem. Have even tried reinstalling git, but the error still exists. Should I simply copy paste the enhancement file from master branch and make the necessary edits? Or should I submit a small patch for the same?

@sumitbinnani I would rather try to fix the git command (you will need it more). Maybe look at http://stackoverflow.com/questions/10125113/git-rebase-is-not-a-git-command-see-git-help

@jorisvandenbossche I am planning to reinstall my OS, as a few more applications/commands are not working correctly after I patched a few packages. However, I am reluctant to do so in hurry owing to data loss, and massive time to get all the packages and libraries required for my job and day to day activities.

Have worked upon the code review and will push the changes after adding more test-cases to avoid multiple commits (given my problem with rebase command).

@sumitbinnani you should just use conda, this makes working with pandas (and all other python packages) very very easy. see the contributing page http://pandas.pydata.org/pandas-docs/version/0.17.1/contributing.html

Hi @jreback, I am using conda only and I am having no issues with python packages and libraries. They are working like a charm. But a few other installation like git have gone haywire and are giving error on otherwise valid commands. I would look into the issue during the long weekend about to come. Meanwhile, I will update the test cases and submit a small patch for the edit in enhancement file.

@jreback: Even the documentation in R have a single use case for the `origin`. However, they have documented a bit about the possible scenarios where it can be used.
ref: https://stat.ethz.ch/R-manual/R-devel/library/base/html/as.Date.html

Does the patch requires any more changes? @jreback? @jorisvandenbossche?

Maybe it would better to have the default be `origin='unix'`, to more explicitly indicate that time is counted from the unix epoch?

It looks like the current implementation is broken unless `unit='D'`. Please add tests and any necessary fixes for other time units.

@shoyer The current implementation is indeed breaking unless `unit='D'`. Would resolve it asap.

can you rebase / update