API: expose pandas.errors
State: closed

closes #14800

this doesn't actually deprecate the existing (as we have discussed in #15181) but baby steps.

I also didn't add the 'hated'! ``SettingWithCopy*`` ones; these will be removed in pandas 2.0 anyhow, so no point in adding them.

Labels:
name: API Design
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@shoyer xref  https://github.com/pandas-dev/pandas/pull/15537#issuecomment-283431033

@jorisvandenbossche xref https://github.com/pandas-dev/pandas/pull/15181#issuecomment-274073773

I think this actually works.

```
In [1]: from pandas.api.exceptions import UnsortedIndexError

In [2]: try:
   ...:     raise UnsortedIndexError()
   ...: except UnsortedIndexError:
   ...:     print("caught")
   ...:     
caught
```

@jreback second one should be `except exceptions.UnsortedIndexError`, ;) Missing the `exceptions`.
thanks @TomAugspurger yes that works too
> I think this actually works.

Yes, but that is because you didn't do any deprecations of the original ones. Referring from two locations to the same exception is no problem, but it was deprecating the old one (eg by using a decorator) that caused the problems discussed before.

I am in favor of putting this just in `pandas.exceptions`
@jorisvandenbossche 

> I am in favor of putting this just in pandas.exceptions

why would you create *another* namespace ? when we already have a perfectly good one.
@jorisvandenbossche can you comment on your objection?
> can you comment on your objection?

Well, that comes back to my initial objections to the `api` namespace :-) (so I should have pressed that more earlier ..) 
I am in favour of removing things of the top-level namespace (functions / sub-namespaces) that people shouldn't use (as we are doing), but in favour of adding things users *may* use.

In this specific case, what I also would do is creating an actual module (let it be `pandas.api.exceptions` or `pandas.exceptions`, that's even a bit different discussion) with the code in there, and let our other modules (in the io code, indexing code, etc) import the exceptions from there (instead the other way around as it is now). That gathers all our custom exceptions in one place (but on the other hand moves them away from where they are used, so that's not necessarily a net win), but also gives a consistent api to users. If they encounter such an exception, it is of the same type (I mean with same class path name) as the one they can use to catch it. And you minimize the chance to 'misuse' a private import.

And to be clear, big +1 to make those exceptions more officially public!
@jorisvandenbossche sure, but these are really 2 separate things.

- an API for exceptions to the user
- move / change the internal usage

I don't think the 2nd part is relevant here. I mean we can always change *how* they are imported into the public namespace. The point here is to have a separate public API layer (just as we did with ``pandas.api.types``. The actual exceptions being defined elsewhere is fine.
I agree with Joris that it would be better to define the exceptions in
`pandas.exceptions` or whereever the public API location is (personally I
like the shorter `pandas.errors`). Unlike __repr__ I don't think Python
provides any builtin way to indicate the canonical location of errors. I
like making a immediately visible location the API location -- it makes it
much harder to unintentionally use the wrong path.
On Fri, Mar 3, 2017 at 8:03 AM Jeff Reback <notifications@github.com> wrote:

> @jorisvandenbossche <https://github.com/jorisvandenbossche> sure, but
> these are really 2 separate things.
>
>    - an API for exceptions to the user
>    - move / change the internal usage
>
> I don't think the 2nd part is relevant here. I mean we can always change
> *how* they are imported into the public namespace. The point here is to
> have a separate public API layer (just as we did with pandas.api.types.
> The actual exceptions being defined elsewhere is fine.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/15541#issuecomment-283993598>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABKS1vdT_M_Eh7sMXWUJz9IsDt1PlGqAks5riDnTgaJpZM4MQFAO>
> .
>

@shoyer @jorisvandenbossche what is the reason to deviate from the established ``pandas.api`` namespace? 

find with ``exceptions`` -> ``errors``. actually was maybe going to split out warnings separately.

This is the point of the top-level api namespace, IOW. 

``api.types``, ``api.errors``, ``api.warnings``, ``api.*`` whatever (imaging ``.algos``, ``.lib``, ``.includes``, ``.src``) etc.

this is going to eventually proliferate. We *cannot* have ``pandas.*`` namespace be *anything* but non-public. 

We have 1 and only 1 public namespace (or at least moving in that direction). 
If the default for namespaces like `pandas.*` going forward is public, what's the advantage of nesting them like `pandas.api.*`?

Most libraries don't use special designated `api` submodules, e.g., consider all the scipy submodules like `scipy.stats`, `scipy.ndimage`, `scipy.linalg`, etc.
no, ``pandas.*`` namespace is NOT public (or at least it shouldn't be). My statement is that is de-facto currently.

I am thinking about ``statsmodels.api``

The point is that pandas NEEDS a specifically public api, that is not just everything the kitchen sink.

just because scipy/numpy does something does not make it the 'correct' way to do things. sure its a nice reference, but pandas has 2 sets of users, API and 'regular' ones. These are different and need to be treated differently.
The problem is that for years people building things on top of pandas have just reached in and used anything. That cannot continue to happen. How better to provide implementation flexibility than to have a specific public API? I thought we had settled this discussion a while back.
Ultimately, I don't think this matters too much, but given a choice between:
1. `pandas.*` indicates private namespace, `pandas.api.*` includes a public namespace, and
2. `pandas.*` includes public namespace, `pandas._internals` (and so forth) indicates private namespace

I think the later (option 2) is more user friendly and consistent with the top level namespace `pandas` already being public facing. In contrast, note that with statsmodels, users only use `statsmodels.api`, not the top level namespace `statsmodels`.

On the other hand, it's true that option 2 is a little harder to transition to (for us), so maybe it's not worth the trouble.

I don't really understand your point about "API" vs "regular" users. I don't think it's a terribly advanced use case to catch appropriate exception types. I do agree that we want to reduce noise in the top level namespace.
> I don't really understand your point about "API" vs "regular" users. I don't think it's a terribly advanced use case to catch appropriate exception types. I do agree that we want to reduce noise in the top level namespace.

I would this is obvious.

- 'regular' users are the ones who pretty much use ``pd.*`` as there namespace. In theory they may use ``pandas.api.exceptions`` (e.g. specifically catching a parser exception).
- 'API' users are using (formerly) pandas internals to build other things on top of pandas. This is the point of an API.

Ultimately these are part of the same namespace that is publicly exposed. The point is that it is completely silly that people can just use *any* of the namespace. Sure you *can*, but pandas has gotten so big, that you simply cannot live that that when you *need* to move things around to change implementation details.

> pandas.* includes public namespace, pandas._internals (and so forth) indicates private namespace

how exactly is this more friendly? I would argue that this is *much* less friendly, simply because you don't know where to find things that you might need. This is a huge surface area, both for dev and for usage. Limiting this scope is hugely important.

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=h1) Report
> Merging [#15541](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/456e729384c315c291ef92bb150bcc4f79a22bdf?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `92.3%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/15541/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #15541      +/-   ##
==========================================
- Coverage   90.98%   90.96%   -0.03%     
==========================================
  Files         143      145       +2     
  Lines       49477    49483       +6     
==========================================
- Hits        45019    45014       -5     
- Misses       4458     4469      +11
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.72% <92.3%> (-0.01%)` | :arrow_down: |
| #single | `40.63% <82.05%> (-0.12%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `94.1% <ø> (ø)` | :arrow_up: |
| [pandas/tslib.py](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree#diff-cGFuZGFzL3RzbGliLnB5) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/lib.py](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree#diff-cGFuZGFzL2xpYi5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/api/lib/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree#diff-cGFuZGFzL2FwaS9saWIvX19pbml0X18ucHk=) | `0% <0%> (ø)` | |
| [pandas/tseries/index.py](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvaW5kZXgucHk=) | `95.43% <100%> (ø)` | :arrow_up: |
| [pandas/io/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsLnB5) | `79.7% <100%> (+0.03%)` | :arrow_up: |
| [pandas/computation/align.py](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree#diff-cGFuZGFzL2NvbXB1dGF0aW9uL2FsaWduLnB5) | `97.91% <100%> (+0.02%)` | :arrow_up: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.52% <100%> (ø)` | :arrow_up: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `90.68% <100%> (-0.29%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.56% <100%> (-0.1%)` | :arrow_down: |
| ... and [15 more](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=footer). Last update [456e729...e5fbdc8](https://codecov.io/gh/pandas-dev/pandas/pull/15541?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
related:

this now happens in our doc builds. 
```
/home/joris/miniconda3/envs/dev/lib/python3.5/site-packages/feather/api.py:43: FutureWarning: pandas.lib.infer_dtype is deprecated. Please use pandas._libs.lib.infer_dtype instead.
  inferred_type = pd.lib.infer_dtype(col)
```

certainly could do a PR upstream to fix this (e.g. with conditional imports). But might be nice
to have
``from pandas.api.lib import infer_dtype``

as a starting location for some library code. This is a useful enough routine that its nice to expose to the outside world.
cc @wesm 
I've been running into these internal API / external API issues lately (e.g. needing to use `DatetimeTZDtype` in an external library), so it might be worth documenting the pandas < 0.19 way to get some of these APIs
We indeed need to document this better.

To come back on the discussion above, I am also more in favor of "option 2" from https://github.com/pandas-dev/pandas/pull/15541#issuecomment-284075240. It's indeed maybe a bit harder to get to that point, but IMO it would be much more clear to be able to say: everything accessible from the `pandas.*` namespace (directly or in a submodule) is public (rather than eg have to say `pandas.types` is private, but `pandas.api.types` you may use). But that is also a more general discussion not only related to exceptions, cfr https://github.com/pandas-dev/pandas/issues/13634




@wesm ``DatetimeTZDtype`` and other pandas extension types are not publicly exposed, API access is only via the ``pandas.api.types`` accessors (just for introspection ATM).

I don't think its time yet to write down a public for any dtype extensions types as the implementation is likely to change in pandas 2 and the only access (aside from the introspectors) is likely to be implementation writers (https://github.com/apache/arrow/pull/375) and not libraries building on top of pandas.

the current API is privately exposed in ``pandas.types.dtypes``
@jorisvandenbossche comment

> everything accessible from the pandas.* namespace (directly or in a submodule) is public (rather than eg have to say pandas.types is private, but pandas.api.types you may use). 

this is how this mess all started. This leads to anyone just reaching in an using any old module (remember ``pandas.core.common``)? and then ties up the implementation by requiring almost internal deprecations when things are moved. The implementation should remain private.

Its clear that *functions* and *classes* from ``pandas.*`` namespace are publicly accessible (we of course have been trying to reduce the vast number here).

Further a fair number of things have moved implementation wise to ``_libs``, implicitly private. But why should ``pandas.tools.merge`` or ``pandas.core.reshape`` be public at all? Its is a PITA to make these all explicitly private (leading _), so that was never done. 

I think a single-one place for API accessibility (and to be honest, we are talking about library *builders* here (on top-of-pandas)., is a really nice idea).

you have ``pandas.*`` and ``pandas.api.*``. Now you want instead to have ``pandas.exceptions`` as well. Where does this stop? we might as well go back to the original, which is very confusing, mixing *some* public and some *private* (maybe non-documented) apis.

The burden should NOT be in the current implementors in pandas. The *possible* API surface is so huge that it HAS to be limited. By default making a very simple. Hey this and this are public namespaces is MUCH simpler, that having a collection of specific namespaces that ARE public and some that are NOT (like ``pandas.exceptions`` and ``pandas.types`` are ok, but ``pandas.tools`` are NOT).

The point and the purpose of the ``pandas.api`` is to provide a flexible method of namespacing the public api.

 For example, how to expose ``pandas._libs/lib.infer_dtype`` (formerly ``pandas.lib.infer_dtype``), which is a useful function for libraries building upon pandas. This clearly belongs on ``pandas.api.lib.infer_dtype``. 

Further I think it would be nice to have ``pandas.api.warnings`` AND ``pandas.api.exceptions`` (of course we would likely import all of these into ``pandas.api`` namespace anyhow, but the point is that is provides *organization* and a single place to look for things.

I imagine that the ``pandas.api`` surface will continue to grow as a public vehicle, while the implementation is simply kept private and can be easily change / improved w/o impacting the user API at all.

Sure one can make an *exception* and put the exceptions in ``pandas.exceptions``, but as I have reasoned above, will just lead to *more* confusion.

@jreback I hear you about `DatetimeTZDtype`, but it turns out that doing zero-copy construction of a DataFrame containing Datetime+TZ data is extremely difficult (I managed to do it, but it was a chore), and uses non-public APIs. I accept that we've made our bed and now have to sleep in it, so at least from 0.20 forward it would be good to not move code or break the "private" API for getting access to DatetimeTZDtype and DatetimeTZBlock 
ok I revised to close #14800. The exceptions are not actually deprecated (as discussed), but I moved all references to the new module ``pandas.errors``.

@shoyer @jorisvandenbossche 
this should be fairly uncontrovertial at this point.
any more comments?
@shoyer well if you want to write some doc strings and copy paste here i will update
Sorry, I don't have the energy for that right now. Let's just make a
follow-up issue(s) -- this could be good for an inexperienced contributor.

On Sun, Mar 26, 2017 at 5:02 PM, Jeff Reback <notifications@github.com>
wrote:

> @shoyer <https://github.com/shoyer> well if you want to write some doc
> strings and copy paste here i will update
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/15541#issuecomment-289327293>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABKS1sj1tpVWP9hOasoZW0LfC8UB4ocgks5rpvx8gaJpZM4MQFAO>
> .
>