BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix crash using to_csv with QUOTE_NONE
State: closed

Fixes #4328.

Now passes `quotechar` to csv's writer. Default is same as csv (`'"'`), but sets to `None` with `quoting=csv.QUOTE_NONE`.


Labels:

Comments:
@jreback can you glance at this before I merge?

looks good

weird csv bug

yeah, the issue is that the default quotechar is `'"'` for csv itself - I wonder whether I can trigger this with normal `csv` writing. Maybe worth filing a bug report...

yep, easy to trigger. Maybe I'll file an issue...

I think this was the wrong solution actually...there's a specific parameter `escapechar` defined that is used only when quoting is set to `QUOTE_NONE`, so we probably should have just added an extra kwarg `escapechar` and used that instead (and pushed the user to make a decision). it's such a minor case, not sure if it's worth revisiting...

Hi, 
I am having this issue still now. My data frame contains unicode strings. Now when I use 'to_csv' without any quoting parameter, I get a csv file where certain fields (2 out of 11 to be exact) are in double quotes. For example:
`28,"May 8, 2013 10:22 AM","76th St, Yates Blvd",41.75717,-87.56633,3,-,1,4,-`
 Now when I use csv.QUOTE_NONE i get the following error:
`Error: need to escape, but no escapechar set`
So i do the following (quoting=csv.QUOTE_NONE, escapechar='"'), and I get this:
`28,May 8", 2013 10:22 AM,76th St", Yates Blvd,41.75717,-87.56633,3,-,1,4,-`
So I get a new quote (") in  those 2 fields. How to get around this problem? Thanks. 

you need to provide a copy-pastable example which reproduces

Please find below:

```
id = [u'1']
date = [u'Jan 12, 2013 08:30 AM']
location = [u'Ashland Avenue, Polk Street']
vehicles = [u'4']
drunken_persons = [u'-']
fatalities = [u'3']
persons = [u'4']
pedestrians = [u'-']

data = pd.DataFrame({'#':id, 'Date':date, 'Vehicles':vehicles, 'Drunken Persons':drunken_persons, 'Fatalities':fatalities, 'Persons':persons,  'Pedestrians':pedestrians, 'Location':location})

data[['#','Date','Location','Vehicles','Drunken Persons','Fatalities','Persons', 'Pedestrians']].astype(str).to_csv('Test.csv',  sep=',', index = False, quoting=csv.QUOTE_NONE, escapechar='"')
```

pls only an example that is not subject to these types of external dependencies. 

OK. I have updated the original post. 

and this is a merged issues, so pls open a new one

Ok, done. Here - https://github.com/pydata/pandas/issues/10783
BUG: series replace should allow a single list
State: closed

Also reverts the `method` parameter of `NDFrame.replace()` deprecation

closes #4743


Labels:

Comments:
@jreback this ok 2 merge? or did u find a `_get_fill_func`-type function elsewhere?

I don't think this will work if you pass odd dtypes eg 'a' to an int series or something ; maybe put some tests that change dtype on purpose 

fyi...may want to move some of your tests to test_generic.py

is it ok if i move in a separate pr?

re strange `dtype`s: what behavior do u want? if you pass a replacement for a value that doesn't exist, then the original series is returned. e.g.,

```
In [16]: s = Series(arange(5.))

In [17]: s.replace({'a': 1, 'b': 1})
Out[17]:
0    0
1    1
2    2
3    3
4    4
dtype: float64

In [18]: s.replace(['a','b'])
Out[18]:
0    0
1    1
2    2
3    3
4    4
dtype: float64
```

but mixing things works

```
In [19]: s.replace(['a',1])
Out[19]:
0    0
1    0
2    2
3    3
4    4
dtype: float64
```

sep PR for tests ok....

what I mean is, just some confirming tests....

```
In [1]: s = Series(np.arange(5))

In [2]: s.replace?

In [3]: s.replace([3],[3.])
Out[3]: 
0    0
1    1
2    2
3    3
4    4
dtype: int64

In [4]: s.replace([3],[3.5])
Out[4]: 
0    0.0
1    1.0
2    2.0
3    3.5
4    4.0
dtype: float64

In [5]: s.replace([3,4],[3.5,'a'])
Out[5]: 
0      0
1      1
2      2
3    3.5
4      a
dtype: object

In [6]: s.replace([3,4],[3.5,Timestamp('20130101')])
Out[6]: 
0                      0
1                      1
2                      2
3                    3.5
4    2013-01-01 00:00:00
dtype: object

In [7]: s.replace([3,4],[3.5,True])
Out[7]: 
0    0.0
1    1.0
2    2.0
3    3.5
4    1.0
dtype: float64
```

note the last one; I am not sure we can do much about that (well we can but only if you use Block.setitem, where I attempt to see if you are using a bool and do the right thing)....but too much work really

ah ok

@jreback this ok?

I guess I don't understand how this works

```
ser = Series([0, 1, 2, 3, 4])
result = ser.replace([1,2,3])
assert_series_equal(result, Series([0,0,0,0,4]))
```

so the 1,2,3 are all replaced by 0? (shouldn't it be `nan`)?

It's treating `1`, `2`, and `3` as "missing" and replacing them by the nearest valid value (this is what the `pad_1d` function does)

ic....so tantamount to `s[[1,2,3]] = np.nan; s.ffill()`

ok then

Yes. In fact, that's what I was doing, but then dtype is not preserved (in cases where it can be)

cool...go ahead then

in test_series.py

```
       def test_replace_mixed_types(self):
(Pdb) l
4697            s = Series(np.arange(5))
4698 
```

This fails on 32-bit because s is int32 (and expected is int64)
use `s = Series(np.arange(5),dtype='int64')` here.....

as the numpy arrays pass thru the dtype :)

I am just going to fix it in a PR I am doing. FYI

ah ok ... cool thanks
ENH ohlc resample for DataFrame
State: closed

closes #2320

Can use ohlc from DataFrame.

cc @jreback

Example:

```
In [24]: df = pd.DataFrame({'PRICE': {Timestamp('2011-01-06 10:59:05', tz=None): 24990,
  Timestamp('2011-01-06 12:43:33', tz=None): 25499,
  Timestamp('2011-01-06 12:54:09', tz=None): 25499},
 'VOLUME': {Timestamp('2011-01-06 10:59:05', tz=None): 1500000000,
  Timestamp('2011-01-06 12:43:33', tz=None): 5000000000,
  Timestamp('2011-01-06 12:54:09', tz=None): 100000000}})

In [25]: df.resample('H', how='ohlc')
Out[25]: 
                     PRICE                           VOLUME              \
                      open   high    low  close        open        high   
2011-01-06 10:00:00  24990  24990  24990  24990  1500000000  1500000000   
2011-01-06 11:00:00    NaN    NaN    NaN    NaN         NaN         NaN   
2011-01-06 12:00:00  25499  25499  25499  25499  5000000000  5000000000   


                            low       close  
2011-01-06 10:00:00  1500000000  1500000000  
2011-01-06 11:00:00         NaN         NaN  
2011-01-06 12:00:00   100000000   100000000 
```


Labels:

Comments:
no what puzzles me is why ohlc fails and describe almost works
(well ohlc is a cython function and describe is not) so there is a disconnect that allows one path to work (almost) and the other to fail

@jreback  What did you think about this one? Not sure what we were looking into re describe (is that a separate issue*?)

\* _describe should have MultiIndex column, rather than index._

can you put a test in for doing the same with describe and see what happens?

When I did this last time and also in master:

```
In [29]: df.groupby('PRICE').describe()  # expected .unstack(1)
Out[29]: 
             PRICE        VOLUME
PRICE                           
24990 count      1  1.000000e+00
      mean   24990  1.500000e+09
      std      NaN           NaN
      min    24990  1.500000e+09
      25%    24990  1.500000e+09
      50%    24990  1.500000e+09
      75%    24990  1.500000e+09
      max    24990  1.500000e+09
25499 count      2  2.000000e+00
      mean   25499  2.550000e+09
      std        0  3.464823e+09
      min    25499  1.000000e+08
      25%    25499  1.325000e+09
      50%    25499  2.550000e+09
      75%    25499  3.775000e+09
      max    25499  5.000000e+09
```

so, it appends it to index, rather than as a MultiIndex column,...

hmm...must be because the `ohlc` is a cythonized and the `describe` is not (so it a general groupby). I think what you show as the ohlc is correct, so then I guess that this a a bug (but different)

so merge? opened issue re describe

ok
CLN: Improve Exceptions in core/frame and testing in test_frame and test_multilievel
State: closed

More specific Exceptions (and better testing for them) through a number of parts of pandas. In particular, in `core/frame` / tests/test_frame and tests/test_multilevel Associated changes in some other parts of pandas to support it (mostly making sparse have the same exceptions as core/frame).
- nearly everything that was a `TypeError` now checks message (to avoid `TypeError` confusion from changes in signature).
- refactored some test methods and changed some places to use `assertRaises` and `assertRaisesRegexp` as context managers.
- core/reshape no  longer uses a specific `ReshapeError` (which wasn't really being used anywhere)
- marked guard test cases with `# pragma: no cover` to hopefully make it easier to use coverage.
- new `raise_with_traceback` method in `compat` which allows for 2/3 compatible preservation of traceback.

Obviously some squashing to come.

Related to #3954.


Labels:

Comments:
This is not a huge change - but anyone want to look over it? 

looks ok to me

the data conflic error in update is odd
maybe make an issue to have someone look at it (I think I understand the rationale but needs a look)

It is weird. I wanted it to inherit from `ValueError` to ease the transition to it (probably) becoming a `ValueError`. Really sticks out (especially because we don't use it _anywhere_ internally).

prob just something vestigial - just change to valueError (which I think u did)?

+1 for `ValueError`...is this used anywhere else?

nowhere. All three uses are: the definition of the exception, in update and in the test cases :P. I'm happy to remove it (right now it's just a subclass of ValueError).

out of there

cut---it---out :scissors: 

okie dokie, goodbye `DataConflictError`. Any other comments?

(raise_with_traceback is pretty awesome. as is this pr)

@hayd thanks! :smile: and thanks for catching those mysterious test removals! I spent some time removing obvious dups but definitely didn't want to remove things arbitrarily!

@hayd I think it's good at this point, if you want to take a final look...

Pretty tricky to spot if you've any other missing, but if you've checked then I'm happy.

Also, re raise_with_traceback please consider answering this SO question: http://stackoverflow.com/questions/17001473/uncatching-an-exception-in-python :)
BUG: Fix DataFrame.from_records w/ normal ndarray.
State: closed

I haven't looked at the `from_records` code before, which is why I'm spending all this time laying out a relatively minor change. Previously, was causing a TypeError about None not being iterable when arr.dtype.names was None. Now goes to 'last ditch' parsing instead.

This was the original test case (that dates back, in one form or another, to at least 2010), looked like this:

``` python
self.assertRaises(Exception, DataFrame.from_records, np.zeros((2, 3)))
```

Problem is that the actual Exception was a `TypeError: None is not iterable`-style exception, not something internal to the code.  I changed this so now an `ndarray` that isn't structured (i.e., has `arr.dtype.names is None`) is treated separately from the structured ndarrays (so it goes into the general case 'last ditch' verison).

Now it basically produces the same thing as passing the array to the DataFrame constructor:

``` python
arr2 = np.zeros((2,3))
tm.assert_frame_equal(DataFrame.from_records(arr2), DataFrame(arr2))
```


Labels:

Comments:
looks good

this is basically a call to main constructor

ot: there is an issue out there for te reverse
eg main constructor should dispatch to from_records if either an ndarray has dtype.names or the dtype is a compound type

okay - let's push that to a separate PR though.

agrees just fyi

Well, this is the kind of thing that surfaces up when you clean up the Exception testing code (plus, it's a lot easier now that you can use `assertRaises` and `assertRaisesRegexp` as contextmanagers...makes it much easier to read).
CLN: Improve Exceptions in core/common, io/excel and core/format
State: closed

Related to #3954.

Cleanups to Exceptions in core/common, io/excel, core/format. A few places I added `# pragma: no cover`, because it looked like the code was generally unreachable. A few cleanups in tests/test_common and better error message in `_mut_exclusive` (which needs to be used in Panel.reindex later on).

Specific places to look at (you can also grep for `# pragma: no cover` in the diff):
- core/common - lines 860 and 1162
- core/format - lines 359 and 823
- io/excel - lines 73, 382


Labels:

Comments:
looks fine
ENH: Better rename/set_names handling for Index
State: closed

Fixes #4628
- MultiIndex now responds correctly to `rename` (synonym for
  `set_names`)
- `set_names` checks that input is list like.
- Cleaner error message for immutable ops.
- Added in previously missing tests for MultiIndex immutable metadata.
- Altered `is_list_like` to check for binary_types as well.

For now, this is the behavior for Index.rename. It's simple and clear and lets existing code just work instead of dealing with typechecks:

``` python
In [2]: ind = Index(range(10))

In [3]: ind = ind.rename((u'A', u'B'))

In [4]: ind.name
Out[4]: (u'A', u'B')

In [5]: ind.names
Out[5]: FrozenList([(u'A', u'B')])
```


Labels:

Comments:
@jreback any comments?

looks good to me
CLN: Remove unused and undocumented kind keyword from read_excel
State: closed

Pops kwarg and warns if you pass it (rather than passing on).

Fixes #4712


Labels:

Comments:
maybe kind was meant to be engine/flavor  at some point to support different backends

I think it might have been for xlsx/xls.

For now, it's just weird to have something in the function signature that's
completely unused, so better to remove it (but, for now, won't error out or
anything if you pass it).

@jreback and others - any objections to merging this?

looks good
BUG: (GH4708) A zero length series written to HDF cannot be read back.
State: closed

closes #4708 by spotting when the sentinel single value has been written. This was not being done for the index which resulted in a values/index length miss-match. 


Labels:

Comments:
can you add a release notes entry under bug fixes, sub-heading HDFStore (which is already there), otherwise looks good

@prossahl you need to handle the empty M8[ns] case; it fails because that case is not caught

@prossahl want to finish this up?

@prossahl ping

Still working on this. Failing Travis builds can not be reproduced locally.

Your test should be this; we don't try to serialize numpy arrays per se
and it DOES fail for me, the datetime64[ns] is not being caught (as empty)

```
    def test_empty_series(self):
        for dtype in [np.int64, np.float64, np.object, 'M8[ns]']:
            s = Series(dtype=dtype)
            self._check_roundtrip(s, tm.assert_series_equal)
```

Travis build is now green with libhdf5 versions 1.8.4 as well as later ones.

- can you add m8[ns] to the tests as well (you convert it but don't test it).
- pls add a one-liner release note (doc/source/release.rst) (ref the original issue 4708), pls put with the other HDFStore bug fixes
- pls squash down commits to 1

and we can get this in

thanks!

thanks for the PR!
ENH: GH3371 support timedelta fillna
State: closed

closes #3371
- ENH: GH3371 support timedelta fillna
- BUG: add support for datetime64 ffill/bfill
- INT: add TimeDeltaBlock support in internals

```
In [3]: s = Series([Timestamp('20130101'),Timestamp('20130101'),
                  Timestamp('20130102'),Timestamp('20130103 9:01:01')])

In [4]: td = s.diff()

In [5]: td
Out[5]: 
0                NaT
1           00:00:00
2   1 days, 00:00:00
3   1 days, 09:01:01
dtype: timedelta64[ns]

In [6]: td.fillna(0)
Out[6]: 
0           00:00:00
1           00:00:00
2   1 days, 00:00:00
3   1 days, 09:01:01
dtype: timedelta64[ns]

In [7]: from datetime import timedelta

In [8]: td.fillna(timedelta(days=3,seconds=5))
Out[8]: 
0   3 days, 00:00:05
1           00:00:00
2   1 days, 00:00:00
3   1 days, 09:01:01
dtype: timedelta64[ns]
```

ffill

```
In [9]: td[2] = np.nan

In [10]: td
Out[10]: 
0                NaT
1           00:00:00
2                NaT
3   1 days, 09:01:01
dtype: timedelta64[ns]

In [11]: td.ffill()
Out[11]: 
0                NaT
1           00:00:00
2           00:00:00
3   1 days, 09:01:01
dtype: timedelta64[ns]
```


Labels:

Comments:
@cpcloud numpy 1.6.1 compat shall be conquered! (for timedelta)....what a mess they made :<

nope that should never even be user visible (the function or the exception)
in reality can just return the value but want it to raise if something slips thru

awesome - just wanted to check. You should add a `pragma: no cover` if you think the assertions should never happen... (that's what I've been trying to do throughout)

@jtratner `timedelta64` and `datetime64` have extraordinarily poor support in numpy 1.6. For example, microsecond units are enforced without regard to...well, anything.

It says [here](http://docs.scipy.org/doc/numpy/reference/arrays.datetime.html#unit-conversion) that _units are not converted correctly_ :unamused:, which has left @jreback and myself looking like this :rage: on several occasions.

[Differences in np 1.6 and 1.7 `datetime64`/`timedelta64`](http://docs.scipy.org/doc/numpy/reference/arrays.datetime.html#differences-between-1-6-and-1-7-datetimes)

@cpcloud that's crazy.

finally got this to pass.....3.2 with 1.6 is of course different in its handing of timedeltas than 2.7 with 1.6....oh numpy
ENH: add PostgreSQL and MySQL support for pandas.io.sql.get_schema
State: closed

Simple changes to add PostgreSQL syntax to pandas.io.sql.get_schema. Added data type cases for get_sqltype() and a columns "flavor" for get_schema(). 

The logic of `get_schema` seems simple enough at the moment (though it still doesn't support Oracle). Should it be replaced by SQLAlchemy integration?

ref: https://github.com/pydata/pandas/issues/4163
old pr: https://github.com/pydata/pandas/pull/4574 


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@adamgreenhall 

yeh....have an open issue to disable that test, kind of flaky; ignore for now

I'm going to cherry-pick this when updating the branch.

I should have suggested before adding some tests for this. (At the moment the legacy tests are quite flaky and it's pretty WET and messy, so this may actually be non-trivial). Saying that, I think it's worth adding this as is too, thanks for putting it together.

_One of the things coming soon (ish) is making it as simple as subclassing the tests with postgres engine/connection, which will be soo much better._

@hayd u r incorporating this?

@jreback yes, but I'm not sure how going to test all this legacy stuff yet (support for different engines is easier), atm no tests for this legacy (aka not sqlalchemy) postgres... I've just left what was there before.

Perhaps solution is to test using the connection against the engine version, not sure how that'll work yet.

@hayd to be honest, if you new stuff is good to go and covers sufficient test cases I would just go with that, maybe even move legacy engines temporarily for 0.13 so they at least 'work' and just migrate to new

@hayd, @adamgreenhall , superceded by #5950?

@hayd @jorisvandenbossche is this superceded? close?

I am not really familiar with this PR, but:
- we decided to not support postgresql in legacy mode, so only with sqlalchemy
- `get_schema` is deprecated

so judging from only the title of this PR (in the diff seems a lot more) this should be closed. 

@adamgreenhall I am going to close, but if you think there is functionality in this PR that is not available in the new sqlalchemy stuff, we can reopen it.
Groupby as index
State: closed

closes #4648
closes #3417
closes #4649 (pr)

```
In [1]: df = pd.DataFrame([[1, 2], [2, 3], [1, 4], [1, 5], [2, 6]], index=list('abcde'))

In [2]: g = df.groupby(0, as_index=False)

In [3]: g.apply(lambda x: x)
Out[3]: 
   0  1
a  1  2
b  2  3
c  1  4
d  1  5
e  2  6

In [4]: g.head(2)
Out[4]: 
   0  1
a  1  2
c  1  4
b  2  3
e  2  6
```


Labels:

Comments:
@jreback also, this fix seems stupidly trivial, but ok to merge too?

I think there may be some more to do on the `as_index` thing (will keep a look out for similar) but this is a good start.

this closes all those? give you a prize of bang-for-the-buck!

go ahead (and stince as_index prob not used too much)..be on lookout
Added Stata 13 support
State: closed

Added initial Stata 13 .dta file format support. Newly added string features ("strls") and writing are not supported. Fixes #4291.


Labels:

Comments:
@PKEuS you have travis turned on?

Yes.

Main error shown on travis is "ValueError: Version of given Stata file is not 104, 105, 108, 113 (Stata 8/9), 114 (Stata 10/11), 115 (Stata 12) or 117 (Stata 13)" for all 117 unit tests, but only for Python != 2.7. Does anybody have an idea what is going on?

the one build that passes runs very few tests (it's for a non-English testing) and doesn't run the stata stuff

does this pass locally?

Travis runs on 32-bit Linux FYI 

Yes, it passes locally. But there is something strange about it I noticed:
When I execute the tests a second time, they fail with the same message as Travis, while they pass, after I changed something in stata.py.

maybe sure that the multiprocess_can_split is False (at the top of the test_stata)

what did u change?

I ran this PR local (64-bit linux) and got this; maybe reading the wrong value for vartypes?

```
(Pdb) p self.path_or_buf
<open file '/mnt/home/jreback/pandas/pandas/io/tests/data/stata1_v13.dta', mode 'rb' at 0x2d5d150>
(Pdb) p seek_vartypes
1441433355735269392
(Pdb) self.path_or_buf.seek(seek_vartypes)
*** IOError: [Errno 22] Invalid argument
(Pdb) p self.byteorder
'>'
```

I was able to fix some problems, now two of 5 buildbots (2.6 and 2.7) are still failing. Now with:

> File "/home/travis/virtualenv/python2.6_with_system_site_packages/lib/python2.6/site-packages/pandas-0.0.0-| py2.6-linux-x86_64.egg/pandas/io/stata.py", line 362, in _read_header
>     self.path_or_buf.seek(seek_vartypes)

Might the problem be that seek_vartypes is read as an 8-bytes integer?

you should be explicit if you are reading bytes
Travis is a 32 bit build

> seek_vartypes = struct.unpack(self.byteorder + 'q', self.path_or_buf.read(8))[0] + 16

Is that not explicit enough? Where do I have to add something? At the line above, or the following one?

> self.path_or_buf.seek(seek_vartypes)

@PKEuS not sure. can you setup a vagrant 32-bit machine and debug?

Finally, the problem seems to be solved!
Endianess was detected wrongly - no idea why it worked with python 3 - apparently python 2 was correct. My code was simply wrong.

@PKEuS great, couple of questions

it looks like this is backward compat (as you can still read the new files). Is there a change in the writing behavior?

can you document in doc/source/release.rst and update the docs in io.rst (I would just add a short blurb that in v0.13 you now accept this new Stata format, while remaining backwards compat)

The StataWriter has not been changed - it still uses the old format.

I will add the new feature to the release notes.

@PKEuS this looks fine

can you rebase on master, then squish down to a few commits....?

Squashed to two commits and rebased.

@PKEuS looks fine to me ... @jseabold ?

I defer to the test suite ;)

@PKEuS thanks again!