BrandonFors/pandas-github-issues huggingface dataset:
BUG: Series.reshape to own shape raises TypeError
State: closed

closes #4554

I just fixed the call to ndarray.reshape with order as a positional argument, which is wrong. Please note that I had to remove a test.


Labels:

Comments:
- can you add release notes entry
- hook up travis (see contributing.md)

@hayd Point taken, I'll try to minimize changes to existing tests.
@jreback Well, there is some choice. We could (a) just reshape the values and always return an ndarray or (b) always return a pandas object if/when pandas can deal with it. I actually like (a).

Back to the code, I realized that more work needed to be done. There are currently two assertions in `test_reshape_non2d()`:

```
x = Series(np.random.random(201), name='x')
self.assertRaises(TypeError, x.reshape, (len(x),))
```

This is the bug I reported. `x.reshape((len(x),))` should not raise `TypeError` but return `x` itself. But I think `x.reshape(s.shape)` makes more sense in this context. I can live with the 201 and the random numbers.

```
# GH 2719
a = Series([1, 2, 3, 4])
self.assertRaises(TypeError, a.reshape, 2, 2)
```

`a.reshape(2, 2)` should return `a.values.reshape(2, 2)` without raising any exception. Asserting this also prevents against #2719.

So I have changed the test according to this and rewrite `Series.reshape()` to pass it. I had to change the signature to make it accept the same input as `ndarray.reshape()`.

Sorry I got git to make a mess of my fork's history. I have been told this can be cleaned up but I am not used to git so I would appreciate some advice.

To cleanup your branch run the following

``` sh
git remote add upstream https://www.github.com/pydata/pandas.git
git fetch upstream
# will need to fix merge conflicts here
git rebase upstream/master
# pops up window, change 'pick' to 'squash' to combine commits
git rebase -i upstream/master 
```

fyi ... you can also replace "pick" with "fixup" to remove the commit messages that squash leaves hanging around. this is great for those commits that fix typos/syntax errors, etc. 

@cpcloud anything else?

@goyodiaz Needs to be rebased.

Sorry, not sure what I'm expected to do. I rebased my changes on top of upstream again.

probably a merge conflict with release notes .... gotta do it again

travis is passing so that's good ... just resolve the merge conflict and BAM! it'll get merged :)

One of the travis build  jobs failed but I don't understand how it can be related to my patch.

It's not crazy that a change to reshape might have caused an issue, though it does seem unrelated. @jreback - thoughts on the failing build?

nope...that's a ocassional test failure that have never been able to nail down (travis once in a while fails it, but never can reproduce locally)....I think this is ready to merge, any objections?

@goyodiaz thank you!
fix generation of short revision hashes in test_perf
State: closed

This PR fixes the broken generation of short revision hashes of test_perf.py described in #4654.


Labels:

Comments:
@y-p ?

Thanks for the ping. I've seen this before, it's fairly rare, due to a hash prefix collision.
That code was a kludge to accommodate vbench's use of abbreviated hashes internally.
Looking at the code again, those come from `git log  --pretty=format:%h` which indeed
corresponds to the "short" hash employed here, rather then to simply truncating. 

cherry picked to master as d53e4fd. Thanks @wuan.
BUG: allow as_index=False for non aggregating groupby methods
State: closed

closes #4648
closes #3417


Labels:

Comments:
@jreback merge-able? pretty trivial fix, but want to get your okay on it.

looks fine

FYI I think there might be an open issue related to this (aside from the one u indicates)

might be the same or unrelelated don't know

Does this work with an index which isn't plain (i.e. not just an arange)? Don't think you're testing for that...

@hayd good call thanks

Does this resolve #3805?

@cpcloud example I had in mind was something like this (with non-boring index), from your branch:

```
In [1]: df = pd.DataFrame([[1, 2], [2, 3], [1, 4], [1, 5], [2, 6]], index=list('abcde'))

In [2]: g = df.groupby(0, as_index=False)

In [3]: g.apply(lambda x: x)
Out[3]: 
   0  1
0  1  2
1  2  3
2  1  4
3  1  5
4  2  6
```

Shouldn't this have index of original DataFrame.

@jtratner nope (just tested)

ah then i misunderstood what `as_index` was supposed to do.

So I think the thing from SO was:

```
In [34]: g = df.groupby(0)

In [35]: g.apply(lambda x: x.head(2))
Out[35]: 
     0  1
0        
1 a  1  2
  c  1  4
2 b  2  3
  e  2  6

In [36]: g = df.groupby(0, as_index=False)

In [37]: g.apply(lambda x: x.head(2))
Out[37]: 
     0  1
0        
1 a  1  2
  c  1  4
2 b  2  3
  e  2  6
```

And we expect the latter not to include the group info.

```
   0  1
a  1  2
c  1  4
b  2  3
e  2  6
```

Now I think about it, we It could be completely misusing apply here...

@cpcloud Think I have an even more trivial fix...!

@hayd Welcome to it! Out of town until Tuesday :)
BUG: (GH4626) Fix decoding based on a passed in non-default encoding in pd.read_stata
State: closed

closes #4626


Labels:

Comments:
@jseabold can you give this a once over; I included your file as a test case, FYI

It appears to me that everything is encoded by the default `cp1252`, the 'encoding' parameter is merely to then decode the strings

cc @PKEuS

pls take a look as well

One comment. It's not a showstopper, but Stata doesn't support unicode. So really the only valid encodings for reading in data are 'ascii' and 'latin-1' or equivalent. Maybe worth a mention in the docs?

so it's really not possible to actually encode the data (unless u somehow did it external to stata?, meaning u encoded the bytes themselves

Yes, AFAIK, the user has no control over this. A great source of anguish to me lately whenever my data passes through Stata. The only extended characters supported are latin-1. Some info here (from third-party software): http://stattransfer.com/faq/encoding.html

ok I'll put in a doc mention
ENH: (GH4629) Support for using a DatetimeIndex/PeriodsIndex directly in a datelike calculation
State: closed

closes #4629

```
In [1]: s = Series(date_range('20130101',periods=5),index=date_range('20130102',periods=5))

In [2]: s
Out[2]: 
2013-01-02   2013-01-01 00:00:00
2013-01-03   2013-01-02 00:00:00
2013-01-04   2013-01-03 00:00:00
2013-01-05   2013-01-04 00:00:00
2013-01-06   2013-01-05 00:00:00
Freq: D, dtype: datetime64[ns]

In [3]: s-s.index
Out[3]: 
2013-01-02   -1 days, 00:00:00
2013-01-03   -1 days, 00:00:00
2013-01-04   -1 days, 00:00:00
2013-01-05   -1 days, 00:00:00
2013-01-06   -1 days, 00:00:00
Freq: D, dtype: timedelta64[ns]
```


Labels:

Comments:
cc @jtratner @cpcloud comments?

Any other index combinations you think I should try?

As an aside, this does not work (nor should it)

```
s = Series(randn(10),index=date_range('20130101',periods=10))
s.index-s
```

as this is an Index set operation (you could of course extract the index from the rhs, but that's not that interesting)

@jtratner ok otherwise?

@jreback sorry about that - can only work on this outside of work (and I just realized that I just misread earlier)

@jtratner np.....I am out of the office next week...so had a really light week this week....got a lot done!

Epic. Thank you much.
now sectionwise: excel / date_parser #4332
State: closed

This is the second part of the fix.


Labels:

Comments:
As requested in #4631 the python modifications are now singled out & separated from the doc changes

Please clarify the title of this PR (I'm not clear how this fits)

closing as too broad. @timmie A PR should address a single item w/o touching any ancillary code. You are free to resubmit if you would like.
now sectionwise: date_converter: delta / time
State: closed

Date converters used for testing the resolution of #4332

See also: #4631


Labels:

Comments:
See comment [here](https://github.com/pydata/pandas/issues/1180#issuecomment-31418130).

Sorry @timmie , I know your PRs haven't had much success with us so far.

@timmie @y-p A suggestion, to not get this lost as this could be interesting for some users, it could maybe interesting to add this to the cookbook (instead of including it in pandas source code)? 
Most items there are links to anther page (mostly stackoverflow), so @timmie, if you could put the example somewhere online, you could add a link to that in the cookbook?
API/CLN: setitem in Series now consistent with DataFrame
State: closed

closes #4312

setitem in a frame that can change dtype
setitem_with_indexer now uses the Block.setitem machinery


Labels:

Comments:
side note - is there a difference between `_maybe*` methods and `_possibly*` methods?

no...i think possibly -> maybe?

@jtratner I know you are going to do a pep/cleanup.....so maybe add that to the list....

@jreback not a big deal, just wondering if I was missing something :smile:
Fixes issue with TextFileReader using python engine and thousands != ","
State: closed

Fixes #4596


Labels:

Comments:
Related to https://github.com/pydata/pandas/pull/4598

I fixed this for floating point numbers in the c parser this weekend. 

IIUC #4598 @guyrt is or the c-parser and this PR #4602 @cancan101 for the PythonParser?

@guyrt your tests will hit both c-parse and PythonParser? (I believe you have many more tests as we discussed)

so maybe will merge yours first

@jreback you understand correctly. My PR has more tests, but they cover the floating point case. This PR covers thousands separators other than ',' in the PythonParser. They could actually be merged in either order.

@cancan101 ok going to merge this, ready?

@cancan101 I realize need release notes, and can you squash?

OK.

I need to go back and do that for a few of my PRs.

@jreback Should be good to merge

@jreback ping

@cancan101 just need a rebase and can merge

@jreback Okay. Should be rebased.

thanks!
BUG: ujson labels are encoded twice
State: closed

With its current handling ujson ends up encoding labels twice, which can cause problems if they contain escapable characters:

``` python
In [16]: df = DataFrame([['a', 'b'], ['c', 'd']], index=['index " 1', 'index / 2'], columns=['a \\ b', 'y / z'])

In [17]: df
Out[17]: 
          a \ b y / z
index " 1     a     b
index / 2     c     d

In [18]: json = df.to_json()

In [19]: json
Out[19]: '{"a \\\\\\\\ b":{"index \\\\\\" 1":"a","index \\\\\\/ 2":"c"},"y \\\\\\/ z":{"index \\\\\\" 1":"b","index \\\\\\/ 2":"d"}}'

In [20]: pd.read_json(json)
Out[20]: 
           a \\ b y \/ z
index \" 1      a      b
index \/ 2      c      d
```

This PR fixes this behaviour so labels are only encoded a single time.


Labels:

Comments:
ok to merge?

Yep, fire away!
TST: fix ujson's intermittently failing date unit test
State: closed

Fixes #4575.

As you surmised @jreback it looks like unnecessary float arithmetic was causing the test failure. I fixed this up and in the interest of consistency removed the dependence on `datetime.now()`

Example

``` python
In [8]: val = 1376774232215504000

In [9]: val / 1000
Out[9]: 1376774232215504

In [10]: val / 1e3
Out[10]: 1376774232215503.8

In [11]: val // 1e3
Out[11]: 1376774232215503.0

In [12]: val / 1000.0
Out[12]: 1376774232215503.8

In [13]: val / 10**3
Out[13]: 1376774232215504
```


Labels:

Comments:
merge?

Yep, good to go. Passes for me on Python 2 & 3

thanks you sir!
BUG: DatetimeIndex uses the wrong union if the operands overlap
State: closed

This pretty much boils down to this line: https://github.com/pydata/pandas/blob/master/pandas/tseries/index.py#L987

You can easily reproduce the error using the following code:

```
r = pd.date_range("2013-01-01", "2013-02-01")
r2 = r + pd.DateOffset(minutes=15)
r | r2 # Results in the index r having "2013-01-31 00:15" appended
```

This happens because the check on the given line does not take into account, that two indices with the same offset do not necessarily have to be aligned. If they are the `_fast_union`-method can be used, if not it should fall back to the `Index` implementation.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
seems easy enough.....PR ?

I'm really sorry for the noise here ...
Travis seems to be angry with me for doing so, but locally all tests run successfully.

can you hook up travis?

Travis seems to work now, the latest commit is already rebased to master.

@filmor thanks!
ENH: add PostgreSQL support for pandas.io.sql.get_schema
State: closed

Simple changes to add PostgreSQL syntax to `pandas.io.sql.get_schema`. Added data type cases for `get_sqltype()` and a columns "flavor" for `get_schema()`. 


Labels:

Comments:
@adamgreenhall take a look at the master issue #4163, also pls hook up travis (see contributing.md)

Thanks @hayd. Will setup Travis and open a pull request against the `sql` branch.

Fantastic! Please ping me when you do. (I've some alchemy work on it pending..)

@hayd: new PR now up at: https://github.com/pydata/pandas/pull/4680
BUG: Fix bug in pd.read_clipboard on windows with PY3 (GH4561); not decoding properly
State: closed

closes #4561


Labels:

Comments:
this is a windows encoding, could use `sys.getsystemencoding()`?

on side note:
cc @jtratner

in pandas/compat.py

```
    def bytes_to_str(b, encoding='utf-8'):
        return b.decode(encoding)
```

shouldn't this be:

```
    def bytes_to_str(b, encoding=None):
        if encoding is None:
             encoding='utf-8'
        return b.decode(encoding)
```

why? strings are immutable so even if it's appended to in the function it won't be seen in the caller

similar to if you increment an `int` parameter

``` python
def f(x):
    x += 1
y = 1
f(y)
print(y) # prints 1
```

also i think it's `sys.getfilesystemencoding()`

@jreback how about: 

``` python
    def bytes_to_str(b, encoding='utf-8'):
        return b.decode(encoding or 'utf-8')
```

For the other one, you can just wrap in an or instead - clearer and doesn't execute get_option if not needed

``` python
encoding = kwargs.get('encoding') or get_option('display.encoding')
```

What's with all of you green icon folk? :smile: 

they must have changed the default, I don't have a pic up

https://travis-ci.org/jreback/pandas/jobs/10216546

I guess pyperclip acts differently on windows and linux under py3.......

it's annoying that all string types in python2 have `decode` and `encode` methods while only `bytes` types in python3 have `decode` and `str` types have only `encode`

you _could_ monkey-patch strings at runtime to make it a noop! but pretty I guess the error means you are not doing something right
Fixes check in _can_fast_union.
State: closed

Fixes #4564.


Labels:

Comments:
can you add:
- mention in Bug for this issue in `doc/source/releast.rst`
- hook up travis? see contributing.md
- I think we need a specific test (e.g. your example) that fails before and is fixed after,
  can put in `tests/tseries/test_timeseries.py`
BUG: Fix Segfault with pd.tslib.get_period_field (GH4519, GH4520)
State: closed

Fixes the following problem (issues #4519, #4520):

``` python
>>> import pandas as pd
>>> pd.tslib.get_period_field(11, 1, 1)
Exception ValueError: ValueError('Unrecognized code: 11',) in 'pandas.tslib._get_accessor_func' ignored
Segmentation fault
```


Labels:

Comments:
Indeed, that would work but it seems to be simpler and cheaper to check f against NULL rather than using Cython's exception specification machinery.

its fine...just pointing out another way...pls rebase and squash commits into 1 and i'll merge you in

Done! Thanks.

@prossahl thank you!
ENH/API: Keep original traceback in DataFrame.apply
State: closed

When "raise <exception instance>" is used inside a try/except block, the
original stacktrace (containing the code path that raised the original
exception).  This makes debugging difficult since all we know is that an error
occured when `frame._apply_standard` attempted to call the provided function.
Preserve the original stacktrace by using "raise" instead of "raise <exception
instance>".  This facilitates debugginb by allowing us to see where (in the
provided callable) the exception occured.


Labels:

Comments:
can you set up travis ci and add a test? thx

I set up Travis CI, but it won't run until I push a new commit (which I'll do once I add a test).  As for the test, do you have any pointers on how one might be able to test that a traceback is the appropriate one?  I'm not sure how one might go about doing this so any tips will be very helpful.

We discussed this before in #4254 (which is talking about the same issue - but didn't get set up for Travis). Don't really need to have a test case for traceback, because it's not really something you can or should test.

right...now i remember. ok then

Ok, in that case how do I force Travis to run since there's nothing new to commit?  Should I delete this pull request/commit and start over?  It looks like Travis is set up OK with my account as far as I can tell.

```
git commit --amend -C HEAD
git push origin mybranch -f
```

will reset the commit hash

Done.  Let me know if you need anything else.  Looks like the build passed.

@jreback @cpcloud this is good (and worthwhile) to merge. And @cpcloud note that it actually fixes the issue you had to mess with to resolve #4403 :grin:

@bburan-galenea I know this is trivial, but can you put a release notes mention ? (so we don't forget when this was actually changed)? thanks...then can merge

Certainly.  Do you mean edit https://github.com/pydata/pandas/blob/master/doc/source/release.rst and add a note under "bug fixes"?

@bburan-galenea yes, that's correct.

Ok, I added the note to release.rst and squashed the commits together.

can you add the original issue in the release notes as well, #4254? which this PR will close

never mind, that is another PR that looks like a dup of  yours (or maybe yours is a dupe of his!) either way...

Thanks!

thank you sir!

keep em coming
DOC/BLD: Update version support in docs and setup.py.
State: closed

Fixes #4412.

Changes install.rst to explicitly reference that pandas supports 2.6,
2.7, and 3.2+. (and removes line that Python 3 support is less
well-tested). Also adds 2.6, 2.7, 3.2, and 3.3 in classifiers in
setup.py


Labels:

Comments:
thank you!
Segfault with pd.tslib.get_period_field FIX
State: closed

closes #4519


Labels:

Comments:
@prossahl
pls do 1 pr per issue
create a new tracking branch off of current master to keep them separate

do 

```
git checkout thisbranch
git checkout -b newbranch
git rebase -i <commit before 1211711>``
git push myfork newbranch
```

and delete the first 2 commits; this makes this branch independent

Abandoned. I'll resubmit this.

@prossahl should this be closed then?

@cpcloud Well without the change the segfault can still happen. I'll submit a new pull request as this one pulled in irrelevant changes.

you don't need to do that. do this

``` sh
git reset HEAD~1
git stash
git fetch upstream
git reset --hard upstream/master
git stash pop
git commit -am'Your awesome commit message'
git push --force
```

Thanks v. much for your help but its a no go.
git stash  results in "No local changes to save". git fetch upstream results in "fatal: 'upstream' does not appear to be a git repository"
I think that my git repo is so messed up it will be quicker to create a new account and start from the beginning...

check out this wiki page:

https://github.com/pydata/pandas/wiki/Using-Git

and also pls read this:

https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md
ENH/API:  GH2578, allow ix and friends to partially set when the key is not contained in the object
State: closed

closes #2578

On a series, this is kind of like an inplace append operation (note the dtype change - on purpose)

```
In [6]: s = Series([1,2,3])

In [7]: s[4] = 5.
In [3]: s
Out[3]: 
0    1
1    2
2    3
4    5
dtype: float64
```

This is now allowed: (previously would `KeyError`), you first have to create the column `C`, then set it

```
In [1]: df = DataFrame(np.arange(6).reshape(3,2),columns=['A','B'])

In [3]: df
Out[3]: 
   A  B
0  0  1
1  2  3
2  4  5

In [4]: df.ix[:,'C'] = df.ix[:,'A']

In [5]: df
Out[5]: 
   A  B  C
0  0  1  0
1  2  3  2
2  4  5  4
```

This would previously raise an `IndexError`

```
In [4]: df.loc[3] = 5

In [5]: df
Out[5]: 
   A  B  C
0  0  1  0
1  2  3  2
2  4  5  4
3  5  5  5
```

and on Panel:

```
In [6]: p = Panel(np.arange(16).reshape(2,4,2),
   items=['Item1','Item2'],
   major_axis=pd.date_range('2001/1/12',periods=4),
   minor_axis=['A','B'],dtype='float64')

In [7]: p
Out[7]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 4 (major_axis) x 2 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2001-01-12 00:00:00 to 2001-01-15 00:00:00
Minor_axis axis: A to B

In [10]: p.loc[:,:,'C'] = Series([30,32],index=p_orig.items)

In [11]: p
Out[11]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 4 (major_axis) x 3 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2001-01-12 00:00:00 to 2001-01-15 00:00:00
Minor_axis axis: A to C

In [13]: p.loc[:,:,'C']
Out[13]: 
            Item1  Item2
2001-01-12     30     32
2001-01-13     30     32
2001-01-14     30     32
2001-01-15     30     32
```


Labels:

Comments:
@jtratner @hayd @cpcloud @wesm thoughts when you have a chance

I guess I wonder what should happen in these kinds of cases cases - any magic?

``` python
df.loc[5] = 3
s[6] = 5
```

looks good.

@jtratner 

these are straightforward

```
In [1]: s = Series([1,2,3])

In [2]: s[6] = 5

In [3]: s
Out[3]: 
0    1
1    2
2    3
6    5
dtype: int64

In [4]: df = DataFrame(np.arange(6).reshape(3,2),columns=['A','B'])

In [5]: df.loc[5] = 3

In [6]: df
Out[6]: 
   A  B
0  0  1
1  2  3
2  4  5
5  3  3
```

I have disabled both `iloc/iat` from being able to enlarge, as would require filling in the intermediary
index with something; which potentially is ambiguous.

```
In [7]: df.iloc[5] = 3
IndexError: iloc cannot enlarge its target object
```

@wesm take a look? this is non-trivial but you were already going it bith set_value in any event, this just generalizes (and with series refactor this is able to be done in-place, which makes pretty natural)

any comments before merging?
DOC: fix conflicting ewma documentation
State: closed

The information in the docs, in the parameter description and in the body of the docstring is conflicting. This should hopefully make them consistent amongst themselves (and with the code).


Labels:

Comments:
Good point! I've updated it so they're all mathified :-)

excellent!

Just noticed #4321 which should be covered by these fixes too (it only updates the docs).

are the docs existing docs just inconsistent with the code (before this change / #4321?)

Yep the docs are inconsistent right now but how they are fixed depends on whether you define `alpha = 1 / (1 + c)` or `alpha = 1 - (1 / (1+c)) = c / (1 + c)`, unfortunately the current docstring and docs have a mix of both of these.

I've made these changes with `alpha = 1 / (1 + c)` in mind as that's what I'm familiar with. Here's the relevant bits from the code (algos.pyx)

``` python
neww = 1. / (1. + com)
oldw = 1. - neww   
...
output[i] = oldw * prev + neww * cur
```

Given the definition for `alpha = 1 / (1 + com)` implies `neww = alpha` and `oldw = 1 - alpha`. Assuming `prev = output[i-1] = y_t-1` and `cur = x_t` means the docs should read

```
y_t = (1 - \alpha) y_{t-1} + \alpha x_t
```

as they do with this fix, and the fix in #4321.

As I said you can do it all the other way around by defining `alpha = 1 - 1 / (1 + c)` but you'd still need to update the docs and docstring to make it all consistent (the docs right now define `alpha = 1 / (1 + c)` but matching the given formula to the code implies `alpha = 1 - (1 / (1 +c))`). The docstring is similarly conflicted.

@Komnomnomnom ok...but your changes make this consistent (with the def of alpha that you are showing)...so that is fine (you might want to mention that it can be done the other way too...., but not go into detail)

Ok I added a very short note to the docs.

@Komnomnomnom merge?
@cpcloud ok?

Yep, go for it. If someone wants to change the wording or anything feel free.