BrandonFors/pandas-github-issues huggingface dataset:
ENH: Preserve .names in df.set_index(df.index)
State: closed

closes #6452.

This causes a slight change in behavior in @jseabold's second example. Previously, `df.set_index(df.index)` would convert a `MultiIndex` into an `Index` of tuples:

```
In [7]: from statsmodels.datasets import grunfeld

In [8]: data = sm.datasets.grunfeld.load_pandas().data

In [9]: data = data.set_index(['firm', 'year'])

In [10]: data.set_index(data.index).index.names
Out[10]: [None]

In [11]: data
Out[11]: 
<class 'pandas.core.frame.DataFrame'>
MultiIndex: 220 entries, (General Motors, 1935.0) to (American Steel, 1954.0)
Data columns (total 3 columns):
invest     220  non-null values
value      220  non-null values
capital    220  non-null values
dtypes: float64(3)

In [13]: data.set_index(data.index)
Out[13]: 
<class 'pandas.core.frame.DataFrame'>
Index: 220 entries, (General Motors, 1935.0) to (American Steel, 1954.0)
Data columns (total 3 columns):
invest     220  non-null values
value      220  non-null values
capital    220  non-null values
dtypes: float64(3)
```

This change makes it so the index remains a `MultiIndex`.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design
name: MultiIndex

Comments:
this is not the way to fix this
instead catch the case of a passed index and treat it like a series rather than the ndarray case

Treating it like a Series doesn't fix the issue described above. What should `df.set_index([df.index, df.index])` return when the index is a MultiIndex? Currently (and in this patch), this would create an Index of two pairs of tuples.

The more correct behavior, in my opinion, would be to return a 4-level MultiIndex. This would require treating the MultiIndex as a DataFrame here or modify `from_arrays` to detect this case (presumably undesirable).

@jreback This newest commit is what I'm thinking. Let me know if there's a more elegant way to get the columns out of a MultiIndex.

@jreback, thanks for the suggestions. Most recent commit has those changes.

looks good. can you add a note to release.rst and v0.14.0.txt both in the API sections, reference this issue and provide a short explanation (prob just a 1-liner - fi you think more is warranted you can do this in v0.14.0.txt with an example - but only if not clear what the change does)

@jreback Added to notes and rebased. Sorry for the delay - I've been sick the last few days.

@jreback Added an example to whatsnew. Let me know if you'd like this branch squashed/rebased.

looks good
pls squash down to 1-2 commits and good to merge 

Alright, squashed and rebased.

thanks!
BUG/TST: sorting of NaNs on sym_diff
State: closed

Closes https://github.com/pydata/pandas/issues/6444, which was failing on Python 3.4

I don't actually have python 3.4 running yet, virtualenv is complaining about my 3.4 executable. These test  _should_ pass on 2.6+.

I put the release note under the bug section, but I can move it to the API section if you think that's more appropriate.


Labels:

Comments:
thanks.....

@TomAugspurger I confirm this is all fixed (on the windows 3.4 build)..thanks!

Thanks for testing it.

On Feb 23, 2014, at 5:31 PM, "jreback" <notifications@github.com<mailto:notifications@github.com>> wrote:

@TomAugspurgerhttps://github.com/TomAugspurger I confirm this is all fixed (on the windows 3.4 build)..thanks!

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/6453#issuecomment-35848054.

np....as an aside...would you be interested in setting up a build-bot type on a mac-mini that numpy hosts? (they can provide a login), you basically setup job that runs (could be a cron) and builds everything..?

I'd be happy to, but I'll probably need help setting it up :) 

see the bottom 2 comments on this thread: https://github.com/numpy/numpy/issues/4328?source=cc

email

rgommers privately and I think he can get you a login....then evaluate whether you think can be done.....thanks

@TomAugspurger also see this: having a whl for numpy would be nice for the mac (then you don't have to compile it!) 

http://numpy-discussion.10968.n7.nabble.com/1-8-1-release-td36603.html
CLN: remove dead code in pandas.computation
State: closed

closes #5041


Labels:

Comments:
Coverage:

```
Name                                 Stmts   Miss  Cover   Missing
------------------------------------------------------------------
pandas.computation                       0      0   100%
pandas.computation.align                96      0   100%
pandas.computation.api                   2      0   100%
pandas.computation.common               14      0   100%
pandas.computation.engines              63      6    90%   94, 122-126
pandas.computation.eval                 62      4    94%   39-40, 46, 90
pandas.computation.expr                330      4    99%   512-514, 538-540
pandas.computation.expressions         101      9    91%   106-107, 140-145, 197
pandas.computation.ops                 231      0   100%
pandas.computation.pytables            350      5    99%   37, 122, 165, 202, 233
pandas.computation.scope               103      8    92%   34-36, 72-74, 135-137
pandas.computation.tests                 0      0   100%
pandas.computation.tests.test_eval    1030      4    99%   50-51, 56, 166
------------------------------------------------------------------
TOTAL                                 2382     40    98%
----------------------------------------------------------------------
Ran 1099 tests in 88.447s

OK (SKIP=26)
```

Most uncovered there are Py3 things

@jreback You ok with this?

Obvs need 2 squash

Alright gonna squash it.

looks fine
BUG/TST: read_html should follow pandas conventions when creating empty data
State: closed

closes #5129
closes #6445

This a very specific bug fix. Only using `lxml` exposes this bug, whereas using
`bs4` raises an exception. `lxml` drops data which allows it to parse the multiindex
header differently and succeed.


Labels:

Comments:
@jreback what do u think about this? this bug makes me think we should change the default flavor to `bs4` because that forces you to say "i'm ok with dropping data" whereas bs4 will keep data raise b/c there's empty data in the header rows

of course that is backwards incompatible .... but only in very few cases i think ... just not sure ... it's annyoing that the two parsers will not parse the same gvien the same data

regardless ... the nan should be changed to the empty string bc that's how the text parser detects empty multiindexes

no idea as I really don't use this

go with consistency if u can

@jreback going to merge after travis passes ... 

sure

Alright I'm going to reopen this, I screwed something up with `git`.
DOC: link to pandas-cookbook instructions
State: closed

Some people were confused about how to use this cookbook, so adding a link to the setup instructions.

see: https://github.com/jvns/pandas-cookbook/issues/7
this SO question: https://stackoverflow.com/questions/21868105/problems-with-pandas-tutorial-missing-file-bikes-csv-and-special-characters-in


Labels:

Comments:
thanks!
DOC: Add section on using in operator with Series, DataFrame
State: closed

closes #5571


Labels:
name: Docs

Comments:
Revised. Thanks, @jorisvandenbossche .

can u add a reference to this section
in basics.rst where series is talked about as dict-like 

thanks

@jorisvandenbossche looks fine to me

I haven't got sphinx working on pandas locally, so I haven't verified this. But the issues discussed above have been fixed.

Looking good (and you can always see the result after a while on http://pandas-docs.github.io/pandas-docs-travis/ to check if everything is OK). Thanks!
ENH #6416: performance improvements on write 
State: closed

- tradoff higher memory use for faster writes. This replaces the earlier PR where the history was a mess!


Labels:

Comments:
@mangecoeur instead of using iterrows.....just do the loop directly; will be much faster as you are not creating a Series (which you then decompose) each time

e.g. just do:

```
        columns = self.columns
        for k, v in zip(self.index, self.values):
          # work with k,v here
```

this munges dtypes btw...e.g. everything gets put into a single dtype. I don't think this matters though?

you _might_ want to consider doing this by dtype, e.g. use df.as_blocks (which you then select the correct column out of each block). the advantage is that they are already dtype separated

@jreback ok i see what you mean. will look into it. 

@mangecoeur this latter approach is used in `to_csv` actually, see here: https://github.com/pydata/pandas/blob/master/pandas/core/format.py#L1285
(as well as chunking the writes to use a constant amount of memory). (so see `CSVFormatter._save`). You can basically do this almost exactly I think. This also has substitution routines for example NaT and such (though may need slightly modifications in `to_native_types` to not stringify (or you might want to do this part inline as you want to return SQLAlchemy types). But you could do something like this in the sql.py:

```
class Block(object):

      def __init__(self, block):
            self.block = block

       def format(self, slicer):
            # return a 'formated block suitable for direct insertion to sql


class DatetimeBlock(Block):

      def format(self, slicer):
          ....

```

etc

@jreback I think we can keep it simple, since the column types are defined on the SQLAlchemy side by the DB table, sqlalchemy deals with converting python values to SQL types already, all we need is to supply a list of row dicts. Maybe we can optimize this more later.

sure....just some thoughts....always profile of course!

@mangecoeur Maybe you can add a vbench to see the effect of this PR? 

@jorisvandenbossche not got vbench working yet. was just from daily use found it was much faster this way :P will get round to it soonish

@jreback had a look at that method, turns out that if you iterate over values you get problems with datetime conversion which using `iterrows` appears to solve. Would need to test on pure numeric data for perf comparison to see if its work trying to fix datetime issue.

@mangecoeur yeh....as I said, the `self.values` call in and of itself bascially munges everything togther. We can merge this and then put up a future investigate perf issue. 

so ready to merge then?

@jreback ok, just cleaned up based on the comments, should be good to go once travis has done its thing.

@mangecoeur thanks!
ENH #6416 improve performance on SQL insert
State: closed

This empirically significantly improves write performance by allowing SQLAlchemy to use the `executemany` method of the driver if available.


Labels:

Comments:
@mangecoeur You have again a problem with the commits (messed up history) .. :-(

A first advice: don't work on master, but create a feature branch for every PR, eg in this case:

```
git checkout -b sql-perf
```

Secondly (but you should do this first, otherwise the history of the branch will also be messed up), to clean-up your history, maybe you can do the following (not fully sure this will work, otherwise @jreback can maybe give some advice):
- save your changes to somewhere else, for example another branch: `git checkout -b sql-temp`
- go back to master: `git checkout master`
- remove all your latest commits you see here in the PR, eg `git reset --hard HEAD~20` for going 20 commits back
- fetch latest upstream: `git fetch upstream`
- rebase your master to upstream: `git rebase upstream/master`
- Now create a feature branch: `git checkout -b sql-perf`
- Now you can try to cherry pick you latest changes (saved in the branch sql-temp) to your new cleaned feature branch: `git cherry-pick <commit>` (with <commit> being the hash of the particular commit you want to restore).
- then you can push this branch to github: `git push origin sql-perf` and create a PR with that.

@jorisvandenbossche yeah i just realized my history is a mess. Am in process of dumping my repo and starting fresh from upstream master.

If you follow my steps above, normally you should be able to easily recover your new commits and cleaning up the history without dumping your whole repo.

But in every case, when you have a fresh repo, don't work on master

I know i shouldn't, was just in too much of a rush to git properly :P anyway, ended up nuking my entire fork and starting again, life is too short for merge conflicts! new PR: https://github.com/pydata/pandas/pull/6420

Looks much better!

There seem to be some more commits in the above list that are not yet in pandas itself?

Closing this PR then.

I think a lot of those commits got squashed when the main SQLalchemy work get merged. That merge history got mangled anyway since it was a bit of a mess, a lot of conflicts that i think jreback took care of.

On 20 Feb 2014, at 14:29, Joris Van den Bossche notifications@github.com wrote:

> Looks much better!
> 
> There seem to be some more commits in the above list that are not yet in pandas itself?
> 
> —
> Reply to this email directly or view it on GitHub.

@mangecoeur Not yet the latest two I think: 'Added interval type' and 'Minor name change'.

@jorisvandenbossche good point, they are actually included in #6420 now. Tests for timeinterval needed. the name changed will probably get overridden by the API design later on anyway.
Make to_csv return a string if no path or buffer is provided.
State: closed

Fixes #6061.


Labels:
name: Enhancement
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO CSV
description: read_csv, to_csv

Comments:
this needs a test to validate that the return value is in fact a string.

I added a test and reverted to the `path_or_buf` argument.

@jorisvandenbossche @cpcloud think we need any more tests on this?

can you add a release note (API section on this)

Done.

thanks!
Remove the special case for empty dataframes.
State: closed

Fixes issue #6062.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
can post a before and after of an empty frame (and also something with rows to make sure it hasn't changed)

Before:

```
print(pd.DataFrame(columns=["asdf", "bsdf"]).to_html())
```

leads to

<table border="1" class="dataframe">
      <tbody>
        <tr>
          <td>Index([], dtype='object')</td>
          <td>Empty DataFrame</td>
        </tr>
      </tbody>
</table>

With the patch you get:

<table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>asdf</th>
          <th>bsdf</th>
        </tr>
      </thead>
      <tbody>
      </tbody>
</table>

A dataframe with values (`pd.DataFrame(columns=["asdf", "bsdf"], data=[[1,2]])`) leads to

<table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>asdf</th>
          <th>bsdf</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td> 1</td>
          <td> 2</td>
        </tr>
      </tbody>
</table>

in both cases.

@filmor no what I mean is actually put up a picture, like a png/jpeg of what it looks like in the browser. You can execute the code and save it that way.

can you add a release note (and a note in v0.14.0.txt for this), referencing the issue? (put in the API section)

looks good otherwise

Done.

@hayd @jorisvandenbossche look ok?

Is this also how it will be displayed when printing a DataFrame in the notebook?

It is, yes. If this is not desired I'd vote for fixing this in IPython, not in `df.to_html`. It makes a lot of sense to have `df.to_html` be parseable again.

thanks
BUG: Fix mergesort unstable when ascending=False (GH6399)
State: closed

closes #6399. I hope I followed all of the correct procedures.

I ran the full test suite (test.sh), and there were no errors.


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
looks good....ping me when travis comes back green

merged via https://github.com/pydata/pandas/commit/8517c05a83899bac0f6f79535723108f7d79df6a

thanks!
CLN: Change assert_(a [not] in b) to specialized forms
State: closed

Work on #6175. Changes instances of assert_(a [not] in b) to specialized assert[Not]In(a, b).


Labels:

Comments:
Busy week; does this need to be rebased?

Is there any easy way on this screen to tell if a PR has conflicts? When creating a PR there is a prominent icon, but I am not seeing anything obvious now that the PR has been created.

If on a PR is say 'this PR can be automatically merged' then no conflicts
BUG: Fix behavior of to_offset with leading zeroes
State: closed

Currently, `pandas.tseries.frequencies.to_offset` erroneously returns a zero time offset when the first part of its argument has a numerical value of zero, even if later parts have nonzero values.  For instance,

```
In [123]: pandas.tseries.frequencies.to_offset('00H 00T 01S')
Out[123]: <0 * Days>
```

This change checks the sign of the numeric part of the argument before converting to `int` in order to support offsets like `'-00H 00T 01S'`.


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
this is fine; can you add a release note (referencing this PR) in Bug Fixes section.

thanks

edit `source/doc/release.rst`

Found it.  Thanks!

I assume I don't have to merge your fix for the test failure into this branch; let me know if I do.

@ischwabacher nope....you _could_ pull master and then rebase (which would pick it up)....but not really necessary as their is no conflict here

thanks for the fix!
CLN: Specialize assert_(np.array_equal(...))
State: closed

Work on #6175. This is the third (and final) "half" of the work started in #6368.


Labels:

Comments:
Sorry for the inefficiency. I had unknowingly changed working directories, so my grep-fu failed me. Now, from pandas/, the following returns no instances of assert_(np.array_equal(...)), so the work started in #6368 is now completed:

grep -nH --recursive -e 'assert_(np.array_equal' .

Investigating the Travis issue; tests had passed locally.

@bwignall that error happens very ocassionaly.....this is fine

thanks!

lmk when you want to close that issue?

Thanks, @jreback. There's still some work to do to get that issue wrapped up, but most of what remains should be pretty straightforward.
CLN: Change assert_(a in b) and assert_(a not in b) to specialized forms
State: closed

Work on #6175. Changes instances of assert_(a [not] in b) to specialized assert[Not]In(a,b).


Labels:

Comments:
Uses new functions added in #6376.

Specialized functions should be drop-in replaceable (including named arguments) for the assertions added in 2.7.

@bwignall another thing that _maybe_ should be done is change the usage of 
`tm.assert_series_equal` and allow `self.assert_series_equal` but that is actually a big change....maybe just make it possible.....?

@jreback, I'll chew on it. That would touch ~1400 lines in the various tests. I will look through them and think about organization.

On the testing docs, @KTAtkinson may be working on that at the moment, as per #6321; if nothing crops up soon, I'll take a crack at it.
BUG: interpolate should preserve dtypes
State: closed

Closes #6290

Replaces PR https://github.com/pydata/pandas/pull/6291 due to github / git issues.

@jreback I think opening a new PR will be easier. I'm was having trouble getting that branch and the PR in #6291 to sync up.

I noticed that you got the missing comma in the join_merge vbench added so I've removed that commit.


Labels:

Comments:
pefect thanks!

minor FYI, always use fully dtypes for comparions...`int` will fail on a 32-bit platform....

As pandas will ALWAYS coerce any input data to int64/float64 (except if its specifically dtyped);
the problem is that on a 32-bit platform is that _numpy_ defaults to int so even something as simple as
`np.arange(5)` will become `int32` on a 32-bit platform

https://github.com/pydata/pandas/commit/173c6860ab653b4be0b4bad4cda122ae5b101ab3
ENH: Add new specialized asserts for tests
State: closed

Work on #6175. Add versions of asserts (assertIs, assertIsNone, assertIn, plus complements) that were added in 2.7


Labels:

Comments:
This is meant to be API-compatible versions of some assertions that were added in 2.7 (http://docs.python.org/2/library/unittest.html), that would be useful for cleaning up #6175.

I am open to suggestions; putting this out as an experiment. Many of the current assert_'s could be specialized to the new asserts that were added in 2.7, but since pandas currently supports 2.6+, I figured it would be easier to add (versions of) the new asserts to the testing module.

@bwignall that's all fine....compat is good!
CLN: Specialize assert_(np.array_equal(...))
State: closed

Work on #6175. This is the second half of the work started in #6368.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
awesome......

and if you have a chance would love to have you enumerate what kinds of test to use when (doesn't have to be long)

as well as any cleanup in here:

https://github.com/pydata/pandas/wiki/Testing

thank you!
PERF: fix performance for series slices (even more)
State: closed

This is a continuation of #6341 and is related to #6370. The latter unfortunately concerns public API and may be not easy to fix gracefully.

So, regardless of the decision to be made in other ticket, this patch will optimize series slicing right away and remove some redundant code (no need to override `__getslice__` because `Series` doesn't inherit `ndarray` anymore).


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Here's the relevant part of vbench output:

```
Invoked with :
--ncalls: 3
--repeats: 3


-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
series_getitem_pos_slice                     |   0.0510 |   4.7594 |   0.0107 |
frame_drop_dup_inplace                       |   2.4463 |   3.0230 |   0.8092 |
frame_reindex_columns                        |   0.3046 |   0.3696 |   0.8241 |
frame_mult_st                                |   3.8100 |   4.3863 |   0.8686 |
frame_mult_no_ne                             |   3.6847 |   4.2224 |   0.8727 |
<snip>
```

looks gr8! pls rebase on master and can get this in

Done.

thank you!
CLN: Specialize assert_(np.array_equal(...))
State: closed

Work on #6175. Work in progress. One more commit needed to finish; splitting to keep reasonably sized.
- Changes assert_numpy_array_equals to assert_numpy_array_equal, for consistency with other methods. (API change of recently-added function)
- Changes instances of assert_(np.array_equal(A,B)) to assert_numpy_array_equal(A,B)
- Update release.rst to reflect work on #6175


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
The proposed API change affects a function @KTAtkinson added this week. I changed all instances in master.

Tests still pass locally.

pls rebase on master - lots of recent changes 

@jreback: should be all set.

@bwignall thanks again!