MAINT: Drop convert_objects from NDFrame
State: closed

Deprecated since 0.17.0

xref #11173

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=h1) Report
> Merging [#15757](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/a9421af1aac906cc38d025ed5db4a2b55cb8b9bc?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/15757/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #15757      +/-   ##
==========================================
- Coverage   90.96%   90.96%   -0.01%     
==========================================
  Files         161      143      -18     
  Lines       49301    49416     +115     
==========================================
+ Hits        44849    44953     +104     
- Misses       4452     4463      +11
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `?` | |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.56% <ø> (-0.25%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.25% <ø> (+3.95%)` | :arrow_up: |
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `93.62% <ø> (+0.18%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `80.85% <0%> (-19.15%)` | :arrow_down: |
| [pandas/tools/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree#diff-cGFuZGFzL3Rvb2xzL3Bsb3R0aW5nLnB5) | `71.77% <0%> (-10.05%)` | :arrow_down: |
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `90.27% <0%> (-9.73%)` | :arrow_down: |
| [pandas/conftest.py](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree#diff-cGFuZGFzL2NvbmZ0ZXN0LnB5) | `94.11% <0%> (-2.32%)` | :arrow_down: |
| [pandas/types/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree#diff-cGFuZGFzL3R5cGVzL2NvbmNhdC5weQ==) | `98.06% <0%> (-1.94%)` | :arrow_down: |
| [pandas/io/pickle.py](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree#diff-cGFuZGFzL2lvL3BpY2tsZS5weQ==) | `79.54% <0%> (-1.71%)` | :arrow_down: |
| ... and [194 more](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=footer). Last update [a9421af...b18e93c](https://codecov.io/gh/pandas-dev/pandas/pull/15757?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Can we hold off on this one until we decide on https://github.com/pandas-dev/pandas/issues/11221 ?

(@gfyoung, sorry, when I saw you were doing all the deprecations, I wanted to raise this before you did this PR, but forgot it)
@jorisvandenbossche you would have to make a clear case for a useful conversion function
lots of arguments against 

not sure it's worth the name either

i would suggest a recipe or expand to_* to a) 2-d b) add a soft option
>  you would have to make a clear case

See my comment in https://github.com/pandas-dev/pandas/issues/11221#issuecomment-145072165 (at that time I was convinced of its usefulness, would have to dig in again for the details), and some of the other reactions. 
But lets keep that discussion there.
@jorisvandenbossche i saw your comment 

reviving a bad api is not going to happen 
you are welcome to propose a new one that has a new name
but that is orthogonal to this issue
@gfyoung what I mean is that [_possibly_convert_objects](https://github.com/pandas-dev/pandas/blob/master/pandas/types/cast.py#L567) should just become part of ``_soft_convert_objects`` (same file), prob rename -> ``_maybe_convert_objects``. This would be much easier once ``convert_objects`` is removed.

This *new* function is pretty much what @jorisvandenbossche is looking for and potentially could be exposed. It allows soft & hard conversions.

I just ran all tests. It seems that *with this PR*. ``_possibly_convert_objects`` is *always* called with the default arguments. I believe its actually called with NO arguments (and hence they are the defaults). except for 1 validation test. e.g.

```
diff --git a/pandas/types/cast.py b/pandas/types/cast.py
index 11a837d..59d0b55 100644
--- a/pandas/types/cast.py
+++ b/pandas/types/cast.py
@@ -568,6 +568,9 @@ def _possibly_convert_objects(values, convert_dates=True, convert_numeric=True,
                               convert_timedeltas=True, copy=True):
     """ if we have an object dtype, try to coerce dates and/or numbers """
 
+    if not (convert_dates == True and convert_numeric == True and convert_timedeltas == True):
+        raise NotImplementedError
+
     # if we have passed in a list or scalar
     if isinstance(values, (list, tuple)):
         values = np.array(values, dtype=np.object_)
```

``pandas.tests.series.test_internals.TestSeriesInternals testMethod=test_convert_objects``
I think it would be worth exposing whatever the new soft convert api is 0.20 (I haven't looked at it in detail), referencing it in the `convert_objects` depr message, then deferring `convert_objects` to the next version, if possible.

I say this because I know there are people (for example, me) who have ignored the `convert_objects` depr message in a couple cases, in particular working with data where you don't necessarily know the columns.  Real instance:

```python
df = pd.read_html(source)[0]  # poorly formatted table, everything inferred to object
                                                # exact columns can vary

df.columns = df.loc[0, :]
df = df.drop(0).dropna()

df = df.convert_objects()
```

Looking at this again, I realize`df.apply(lambda x: pd.to_numeric(x, errors='ignore'))` would also work fine in this case, but that wasn't immediately obvious, and I'm not sure we've done enough handholding (for lack of a better term) to help people transition.
@chris-b1 Can you comment that in https://github.com/pandas-dev/pandas/issues/11221 ?
@jorisvandenbossche @chris-b1 : You both raise some valid arguments regarding the keeping of `convert_objects`, so then I guess we have to back to @bashtage and ask it was deprecated in the first place :smile:.  I am perfectly fine with holding with regards to removal, though there were some minor doc fixes that I would still like to push through at minimum.  How does that sound?
@gfyoung 

it was deprecated because of the magic "if it has 1 valid value convert it, but if nothing converts then revert it" (maybe should make a song!)

So it becomes data dependent, not a great thing for a default. Sure it could be an *option* to the user which would be fine.

This further was exacerbated because the conversion would first try to convert to datetime, then if that failed to timdelta, then to numeric. Even more magic here.

So certainly we could add this back, but it would have to be an explict user decision.
this is basically what it does on top of the current functions: https://github.com/pandas-dev/pandas/issues/11221#issuecomment-246120793
My opinion of `convert_objects` hasn't changed.  As @jreback said, it has a lot of *magic* and the precise behavior in all situations could only be described by the actual code being executed. This does not strike me as good design. 

The real issue here is that the well-defined rules for converting objects have already failed by the time that `convert_objects` was usually called (e.g., text data was imported from `read_`* and `object` was returned).  Given that there are not nearly lossless choices available, it seems very difficult to arrive at a reasonable set of rules that don't seem arbitrary.

If there was a need for a `convert_objects`-like function, then I think it would need to start from scratch by first describing a concrete set of actions that are more aggressive than the default converters but not so as to destroy the utility of this function.   


@jreback : `_possibly_convert_objects` is in `cast.py`, which is not affected by this removal.  The functions which had the `TODO` are in `core/internals.py`  I think your proposed refactoring is beyond the scope of this removal, though it can implications on how to rebuild the API afterwards.
@gfyoung btw, no reason for you to keep rebasing this, unless something is actually changing.
Yeah, sorry, I have been making changes locally, but I haven't pushed anything yet.
np by all means if u r changing things
@jreback : Is there anything else that needs to be changed here?  Even your proposed <a href="https://github.com/pandas-dev/pandas/pull/15757#issuecomment-288066093">diff</a> has already been incorporated into `master` already it seems.
i think this is fine

even IF we want to have a future conversion function could always bring back the name 

i am
+1 in merging 
I am -1 on merging this before a resolution on https://github.com/pandas-dev/pandas/issues/11221
@jorisvandenbossche : Could you clarify as to what a resolution would entail in #11221?  Also, I think @jreback had some questions that have yet to be answered about it.
@gfyoung no need to rebase this until we decide :>
i am in favor of merging this. No action has happened on the replacement since 0.20.0. If it is needed, someone can open an issue and/or implement this. If really really needed (which IMHO is not necessary anyhow). 
@jreback : Don't forget that #11221 (which you opened regarding `convert_objects`) is still active.  So we also redirect people to re-open the discussion at a later point.
Also, if you do want to merge this in earnest, let me know, and I'll turn on testing to get the green light.
@jorisvandenbossche I don't think waiting on this is healthy. We haven't had any activity w.r.t. #11221. if someone really needs this then a new API can be designed. 

@chris-b1 
@jreback : I might also point out that you had mentioned moving forward on this PR about a month ago <a href="https://github.com/pandas-dev/pandas/pull/15757#issuecomment-306157021">here</a> with no response.

Thus, I am also in favor with merging this AND closing the related issue, which can always be revived in the future by anyone who's interested.
yes, let's close this and proceed as indicated in #11221, create ``infer_object`` and change the deprecation message on convert_objects (and schedule for next release). as indicated https://github.com/pandas-dev/pandas/issues/11221#issuecomment-314934018