API: map() on Index returns an Index, not array
State: closed

- [x] closes #12766
- [x] closes #12798 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

This is a follow on to #12798.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design

Comments:
Note that the following changes result in some additional changes to the API which should potentially be addressed so I would appreciate suggestions.
- MultiIndex maps to Index: because mapping a function can reduce or increase the dimensionality of the index I am returning an Index from these operations rather than inspecting the data to determine whether the output should potentially be a MultiIndex
- Categorical and CategoricalIndex: When a categorical cannot be returned returns an Index of the appropriate type rather than an np.ndarray 

@nateyoder use `ensure_index` to sniff for Index/Multi_index. You can simply return a `CategoricalIndex` (rather than a plain `Categorical`).

This will need to be in 0.20.0

@jreback I may have misunderstood what you were saying but I gave using _ensure_index a shot with the below code but it didn't seem to do anything.  I believe this may be because the map returns a np.ndarray rather than a list which would then be converted to a MultiIndex?  Should I go ahead an put a tolist in there despite the performance drawbacks or am I totally barking up the wrong tree? 

```
_ensure_index(Index(self._arrmap(self.values, mapper), **attributes))
```

hmm, actually it IS prob worth introspecting this and if it _is_ ndim==1, then its no problem, if its is an array-like of tuples, then you can pass to `MultiIndex.from_tuples` directly.

Because the outputs of _arrmap were actually always contained in a 1d np.ndarray I actually checked for the presence of a tuple.  Let me know if you'd prefer something else.

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14506?src=pr) is 85.31% (diff: 100%)
> Merging [#14506](https://codecov.io/gh/pandas-dev/pandas/pull/14506?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #14506   diff @@
==========================================
  Files           144        144          
  Lines         51016      51022     +6   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43522      43528     +6   
  Misses         7494       7494          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [3ba2cff...95e4440](https://codecov.io/gh/pandas-dev/pandas/compare/3ba2cff9c55cd16b172f9feb09da551990753f3b...95e4440d18b499c79e1ae5de81af0bb0d86f7eae?src=pr)
Great catch on the DatetimeIndex!  Sorry I missed that. I'll try to update the PR today.

Unfortunately when making this change I ended up causing two other tests (series.test_apply.TestSeriesApply.test_apply_datetimetz and series.test_apply.TestSeriesMap.test_map_datetimetz) to fail because they became np.int64 instead of np.int32.  Is this a significant failure that I should try to work around or should I change the dtype in the tests?

Just wanted to check and see if there were any additional actions you would like taken on this PR?  It appeared to me as though the AppVeyor failure was not due to my changes but am not familiar with AppVeyor.

@nateyoder sorry for the delay. this looks pretty good. couple of comments. Pls have a look at the dtype changes w.r.t. `.apply(lambda x: x.hour)`; these _should_ be `int32`.

@sinhrks thoughts

Hi all.  Apologies for the delay on this after your thorough review.  I've updated the documentation and I think I've made almost all the changes you asked for.

However WRT the int32 vs int64 case I ran into a bit of an issue.  This seems to be happening because I need to return an index when map is applied to and index and current behavior is to force any numeric data type to the classes default rather than allowing the user to specify the dtype.  This can be seen here: https://github.com/pandas-dev/pandas/blob/master/pandas/indexes/numeric.py#L41 where the input dtype is completely ignored.  I can go ahead and make the change to use the input dtype but it would result in some potentially confusing situations where an Int64Index has a dtype of int32, int16, or even int8.  It would also of course impact other tests.  I've already started on these changes but before charging ahead thought I should probably see what you'd like to do.

Thanks!
@nateyoder construction of an Index with a particular kind of dtype is not allowed, e.g. we cannot have int32's in an Int64Index, only int64s. So you infer the best you can but the constructors will promote if needed; you can actually construct like this:

So it will respect the flavor but not the itemsize. This is no purpose.
```
In [1]: Index(np.array([1,2,3]),dtype='int32')
Out[1]: Int64Index([1, 2, 3], dtype='int64')

In [2]: Index(np.array([1,2,3]),dtype='float32')
Out[2]: Float64Index([1.0, 2.0, 3.0], dtype='float64')
```
Hi @jreback.  Sorry for my confusing message.  I understand that it is the current behavior of indices to ALWAYS return `int64` but if you would like to be able to return `int32` rather than `int64` types from map operations on Indices as you had said two weeks ago I believe that would have to change.  Would you like me to try to make those changes or are you alright with map operations always returning `int64` if an integer type is returned?  Apologies again for the confusion.
not sure what you are referring
@jreback I think the confusion is this: you ask that the return dtype *should* be int32 (as before: return type of an int32 array), but for an Index this can never be the case as even an int32 array will result in a int64 index.

So for `Index.map`, the resulting dtype *has* to change in this PR.

The question remains of course if it is possible to keep the dtype of `Series.map/apply`. But, this is already a bit inconsistent at the moment:

```
In [45]: dtidx = pd.date_range("2012-01-01", freq='H', periods=5)

In [46]: dtidx.map(lambda x: x.hour)
Out[46]: array([0, 1, 2, 3, 4], dtype=int32)

In [47]: dtidx2 = dtidx.tz_localize("Europe/Brussels")

In [48]: dtidx2.map(lambda x: x.hour)
Out[48]: array([0, 1, 2, 3, 4], dtype=int32)

In [49]: pd.Series(dtidx).map(lambda x: x.hour)
Out[49]: 
0    0
1    1
2    2
3    3
4    4
dtype: int64

In [50]: pd.Series(dtidx2).map(lambda x: x.hour)
Out[50]: 
0    0
1    1
2    2
3    3
4    4
dtype: int32
```

So for a series it gives in64 for datetimes, but int32 for timezone aware datetimes. So personally I wouldn't mind of all those case become int64 (for Index.map it will be in any case, only leaving tz aware Series to return int32) 


@nateyoder @jorisvandenbossche response is good (I dont think this is a big deal that we r actually returning int32 from the datetime computations - can evaluate that on another issue)

I think my original comment was why we needed to change a test from int32 to int64; since we are always returning an Index which by definition have restricted dtypes the test needs to change

so ought to document this with an example of that (in whatsnew)
Sounds good.  I updated the whatsnew and believe I have no addressed all of the comments.
@nateyoder Could you update for the latest comments of @jreback? Would be nice to get this in!
@jorisvandenbossche Sorry for the delay.  Let me know if you notice anything else that needs to be updated.

Thanks I've enjoyed my first contribution!
lgtm. just a small doc change.
thanks!

great effort!
@nateyoder Thanks a lot!