BrandonFors/pandas-github-issues huggingface dataset:
Respect byteorder in StataReader
State: closed

Partially fix #11282 for s390x


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: IO Stata
description: read_stata, to_stata

Comments:
would love to include, but can you include a test. ideally write a test frame in (both little/big) endian then read back.

Actually I detected that because of this test:

nosetests -s pandas/io/tests/test_stata.py:TestStata.test_read_dta18

This test was OK in x86_64 and wrong on ARM and S390X machines, now is green in both.

don't delete tests
better to skip that on platforms where it is inappropriate (and use a different one where it is)

so ideally have a file in a different byte order then test it can be read in the correct platform (and errors on incorrect platforms)

Does anyone have access to Stata on a BE machine?  If so we should just convert a few of the test files to BE and then test that we can read them back in on a LE machine.

I don't have access to a BE machine with Stata, but I can provide Stata code to load/save the files.  If run on a BE machine that should convert the file byteorder to BE.  I don't think Stata will save files in non-native byte order, so I can't do this on any of my machines.

closing, but pls reopen if you can add tests
added pd as namespace
State: closed

more consistent with code in the rest of the document


Labels:
name: Docs

Comments:
@dacoex yeh, you will need to explain why you want to add a warning like that (in another PR). 

@jreback seems that you are not convinced of the usefulness.

I am talking about the keys accessor warning. just take out for now and you can put in another PR. (e.g. just the `pd.` changes here.

No I am not convinced its something people should be doing at all. The attribute access is just a convenience feature, string accessing by paths is the preferred method.

OK, now this PR is only about the namespace changes.

can you rebase this

pls don't open new pull-requests for the same fix, just force push to the same. see the [docs](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

simply

`git push yourbranch yourremote -f`
PEP8: fix sql.py and test_sql.py
State: closed

Master is failing due to my PR https://github.com/pydata/pandas/pull/12205

Strange that it did not fail on travis before merging ?


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
it was failing the linting before but that was turned off until very recently to actually fail Travis 

I usually check these before I manually merge but didn't in this case
BUG: Make Styler ignore None index names to match to_html behaviour
State: closed

Addresses https://github.com/pydata/pandas/pull/12090#issuecomment-180695902 by making the `Styler` behaviour match regular `to_html` behaviour. 

This PR is based from #12162.
##### New behaviour

<img width="596" alt="screen shot 2016-02-08 at 10 35 23 am" src="https://cloud.githubusercontent.com/assets/3064019/12890011/b183e81e-ce4f-11e5-9b9f-c021fcb33c5a.png">

cc @TomAugspurger 


Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Actual commit: https://github.com/pydata/pandas/commit/0cb2be1cd94483057b59bc714b0a73a7bfe27b77

Tests are failing on lint with files this PR doesn't touch: https://travis-ci.org/pydata/pandas/jobs/107799224#L1134-L1154

```
$ ci/lint.sh
inside ci/lint.sh
discarding /home/travis/miniconda/bin from PATH
prepending /home/travis/miniconda/envs/pandas/bin to PATH
linting -> pandas/core
linting -> pandas/io
pandas/io/sql.py:24:1: F401 'Appender' imported but unused
pandas/io/tests/test_sql.py:2327:33: E127 continuation line over-indented for visual indent
pandas/io/tests/test_sql.py:2675:33: E127 continuation line over-indented for visual indent
pandas/io/tests/test_sql.py:2741:25: E127 continuation line over-indented for visual indent
linting -> pandas/stats
linting -> pandas/compat
linting -> pandas/sparse
linting -> pandas/tools
linting -> pandas/tseries
linting -> pandas/tests
linting -> pandas/computation
linting -> pandas/util
The command "ci/lint.sh" exited with 1.
```

Joris is fixing that [here](https://github.com/pydata/pandas/pull/12263). I'll restart your travis build once that's merged.

@HHammond can you rebase on master when you get a chance. I _think_ it should be a clean rebase.

@jreback Wanted to double check with you before I mess anything up, the merge script will handle this merge cleanly right? There's the two commits, one of which is now in master. Second commit has no conflicts.

you can run the merge script then pause when it asked if u want to push upstream
thrn in another window check that everything is good (you can even edit the commit message at this point)

OK, thanks. I'll document that in the wiki.

For some reason, the travis builds started to fail after merging this one. But does not seem to be related at first sight ("ImportError: libgfortran.so.1: cannot open shared object file: No such file or directory" and "ERROR: test_get_all_data (pandas.io.tests.test_data.TestYahooOptions)")

Looking into it now, sorry.
DOC: pd.read_csv doc-string clarification #11555
State: closed

closes #11555 
Updated IO Tools documentation for read_csv() and read_table() to be consistent with the doc-string,
also reordered keywords to group them more logically. Also updated merging.rst docs for concat.


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
Migrated from https://github.com/pydata/pandas/pull/11756

Here's what the IO docs would look like:

![screen shot 2016-02-07 at 9 30 19 pm](https://cloud.githubusercontent.com/assets/5619798/12879722/e10f86d2-cdec-11e5-833e-ac1d5f2d061f.png)

looks like a couple of linting errors

```
linting -> pandas/io
pandas/io/parsers.py:51:80: E501 line too long (80 > 79 characters)
pandas/io/parsers.py:54:80: E501 line too long (80 > 79 characters)
pandas/io/parsers.py:62:80: E501 line too long (80 > 79 characters)
pandas/io/parsers.py:67:80: E501 line too long (80 > 79 characters)
```

Fixed long line lint errors.

ok, ping when green.

I think we can merge this.

@jorisvandenbossche ok ?

@frankcleary actually, pls add a note in whatsnew, maybe in Enhancements (with a link to the docs)

thanks @frankcleary 

when docs are built (prob a few hours), have a look: http://pandas-docs.github.io/pandas-docs-travis/

and if needed pls issue a follow up to correct

@frankcleary looks good!

@jorisvandenbossche 
was wondering if we should put references (this for io.rst) in the 'doc-string' area to some of the examples
e.g. http://pandas-docs.github.io/pandas-docs-travis/io.html#na-values could be linked from the `na_values` option.

?
BUG: Ensure data_columns is always a list (i.e. min_itemsize can exte…
State: closed

…nd it)

closes #10381


Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
can you rebase/update

can you rebase / update & add whatsnew for 0.18.2

can you update according to comments

can you rebase / update?

Yeah, I'm just trying to focus on [one PR](https://github.com/pydata/pandas/pull/13205) at a time to limit (my) confusion... but this will be the next.

sure no prob

can you rebase / update?

Not next week, I fear, but yes the one after.

closing, but pls reopen / comment if you can continue

See #14728 (sorry for the long wait)
DOC: strings longer than min_itemsize are not truncated
State: closed

Fixed docs on min_itemsize


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Sorry, closed by mistake

this is just a doc change, but you are wanting to change the error, no?

No, my intention (here) was indeed just to update the docs.
(The other open PR related to `min_itemsize` has not much to do with this.)

(I think the current behaviour is the sane one, don't you?!)

ok why don't u update the doc string of append with an example as well then

thanks!
Hdffixes
State: closed

This fixes #12221 and #10381 . Same source file, but let me know if you want separate PRs.


Labels:

Comments:
these changes are separate

the iterable a ones don't have tests

Yes, they are separate: do you prefer separate PRs?

The iterable one has test at line 394 of `pandas/io/tests/test_pytables.py`.

separate

the iterator one needs to inherit from BaseIterator

OK, now repushing them separately. But BaseIterator is not a meaningful ancestor for HDFStore itself, since the latter can be iterated more than once, i.e. doesn't have state (i.e. doesn't have `__next__`).

Replaced by https://github.com/pydata/pandas/pull/12251 , https://github.com/pydata/pandas/pull/12252 , https://github.com/pydata/pandas/pull/12253
TST: allow assert_almost_equal to handle pandas instances
State: closed

Closes #11584.

Because many existing tests expects `assert_almost_equal` to compare `pandas` instances, it looks better to call appropriate test functions rather than raise. 

I've also fixed some failure cases caused by the change (comparing `Series` and `list`, etc).


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
just a small question
TST: Skip scipy NaN test on 0.17 for now
State: closed

Working around https://github.com/pydata/pandas/issues/12240 which we'll leave open until I (or someone more knowledgeable) can properly fix these.

This skips the following tests when on scipy 0.17
- test_nansem
- test_nanskew
- test_nankurk


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @TomAugspurger
BUG: Timestamp subtraction of NaT with timezones
State: closed

Closes #11718.
- Ops with `NaT` and `Timestamp` with `tz` results in `NaT`
- Ops with `NaT` and `DatetimeIndex` /  `TimedeltaIndex` results in corresponding `Index` filled with `NaT`.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timezones
description: Timezone data dtype

Comments:
@sinhrks one small comment. looks much cleaner. thanks. merge when green (or ping and I can).

Thanks, fixed to use `ABCPeriodIndex`. Will merge when green.

lgtm!
TST: fix some scipy 0.17.0 changes
State: closed

partially on #12235 


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
this doesn't fix some of the `test_nanops` breaks in `kurt/skew/sem`

It'd be nice to have comments explaining why we need the conditional tests. Reading through the scipy release notes, I don't see why the interpolate test would have changed, but they don't list bug fixes.

This + https://github.com/pydata/pandas/pull/12243 should get us passing again.
DOC: Remove claims of unbiasedness from doc string for std
State: closed

xref #12230


Labels:
name: Docs
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@TomAugspurger ?

+1, For reference the `np.std` docstring also clarifies that a `ddof=1 still won't be an unbiased.

is it worthwhile putting in a link to the wiki on the doc-string?

Do you mean our wiki or wikipedia? I'd say no either way probably.

Are you OK with merging even though that py35 build failed? I saw you open that other issue.

Actually should I just say 'sample standard deviation' in `basics.rst` and `computation.rst`, since the correction can be specified by passing `ddof`?

Well actually the same can be said of variance, so never mind.
Read very old stata DTA files
State: closed

This should close #12232, although the issue may resurface for files containing double values (I can't determine the old type code for doubles).


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: IO Stata
description: read_stata, to_stata

Comments:
@kshedden looks good.

can you add a release note and fix flake8 issues (that's why the build failed)

looks good. minor flake8 issue to fix & pls add a release note & squash. ping when green.

@jreback it's squashed, etc. but I get what seems to be an unrelated test error (numexpr) on only one platform

@kshedden thanks!
Use correct prop_cycle in the mpl_stylesheet.
State: closed

Fixes https://github.com/pydata/pandas/issues/11727


Labels:
name: Visualization
description: plotting
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
is this independent of #12190? 
pls add a whatsnew note as well (bug fixes is good)

#12190 is deprecating it, but I suppose it doesn't hurt to fix it before we deprecate :)

thanks!
BUG: Strings like '2E' are incorrectly parsed as valid floats
State: closed

A work colleague, David Chase, encountered some surprising behaviour, which can be reduced to the following.  The data-frame

```
DataFrame({'x': [2.5], 'y': [42], 'z': ['2E']})
```

does not round-trip correctly.  The string '2E' is interpreted as a valid float, but it should not be (according to man strtod(3), which seems a reasonable spec).

This PR changes the three variants of `xstrtod()` to reject a string where no digits follow the 'e' or 'E', and includes tests for this case.


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks for feedback.  Will tidy up the following:
- move test to `io/test_parsers.py`
- use `self.read_csv` in test
- remove redundant assertion on behaviour with `'2E3'`

and force-push new commits.

New commits address feedback from @jreback.  Can squash or otherwise re-order if preferred, but for review, separate commits seemed cleaner.

pls add a whatsnew note (bug fixes)

Have created an issue (#12237), to have something to refer to in the whatsnew entry.

@bennorth you didn't actually have to create an issue as you can just refer to the PR number itself (but ok)

(Ah, OK, sorry.  I saw that the other entries in whatsnew did refer to issues so followed that pattern.)

I should have seen that one --- there is no `self` in that top-level function.  Will revert.

(Build/test failure matches failure in current `master` and appears unrelated to this change.)

ok, couple of small comments. pls rebase on master & squash. ping when green.

(Rebased on master and squashed.)

(Force-push of b51a35a is fresh re-base onto master.)

@jreback Pinging as requested --- squashed, rebased, and build is green.

@bennorth lgtm. But have a look here: https://github.com/pydata/pandas/pull/10133

maybe move your tests near this one (if they are not there now)

sorry, rereading this had to do with the decimal places in the mantissa I guess.

Yes, it does make sense to move the new tests near related ones.  New commit on the way.  Thanks.

Recent build failures were in `master` and unrelated.  Have just force-pushed newly-rebased commit.

thanks @bennorth 

Thanks!
STYLE: final flake8 fixes, add back check for travis-ci
State: closed

closes #11928


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
So this is all fixed, but I can't get this script to fail on flake errors.

https://travis-ci.org/jreback/pandas/jobs/106597193

anyone know what is wrong here? in `ci/lint.sh`
ENH: formatting integers in FloatIndex as floats
State: closed

Ref issue #12164

Previous Behavior:

``` python
   In [2]: s = Series([1,2,3], index=np.arange(3.))

   In [3]: s
   Out[3]:
   0    1
   1    2
   2    3
   dtype: int64

   In [4]: s.index
   Out[4]: Float64Index([0.0, 1.0, 2.0], dtype='float64')

   In [5]: print(s.to_csv(path=None))
   0,1
   1,2
   2,3
```

New behavior

``` python
   In [2]: s = Series([1,2,3], index=np.arange(3.))

   In [3]: s
   Out[3]:
   0.0    1
   1.0    2
   2.0    3
   dtype: int64


   In [4]: s.index
   Out[4]: Float64Index([0.0, 1.0, 2.0], dtype='float64')

   In [5]: print(s.to_csv(path=None))
   0.0,1
   1.0,2
   2.0,3
```


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Clean

Comments:
did this break anything non-trivial?

Tests fails with Python 3.5 because I'm an idiot. I'll fix that tomorrow (the test failure I mean, because me being an idiot will take more time to be fixed)

Otherwise, no, it doesn't seem to have broken anything non-trivial.

@jreback all green
Add ability to specify AWS s3 host through AWS_S3_HOST environment variable
State: closed

The AWS s3 host used by read_csv currently defaults to s3.amazonaws.com and there is currently no way to change this. This pull requests adds functionality where the code checks the AWS_S3_HOST environment variable for a user-specified host and defaults back to the current s3.amazonaws.com host. 

This is a simple code change, but it will allow users from around the globe to use read_csv to directly read from files in s3 when the s3 bucket is not located in the US East region.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Can this be [configured through boto](http://boto3.readthedocs.org/en/latest/guide/configuration.html#configuration-files)? I think `~/.aws/credentials` can have a `region` key. It'd be nice to use their config system rather than trying to work around in pandas.

Unfortunately, it can't. You can't add it as an option to the .boto configuration either. [Other packages](https://github.com/jschneier/django-storages/issues/28) have implemented a similar work-around. 

@mlurie ok, thanks for this

see #12206 for adding more docs w.r.t. s3 and options that we support
CLN FloatArrayFormatter
State: closed

I'm working on #12164

This is a work in progress. 


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Clean

Comments:
Ok, that should do.

What I have done:
- Instead of differentiating on the one hand `FloatArrayFormatter.get_result` (which comes from `GenericArrayFormatter`), and on the other hand `FloatArrayFormatter.get_formated_data`, according to their destination (two usages, two functions, two independent pieces of code), I differentiate with the type of the output: `.get_result` keeps returning a list like before, and the new `.get_result_as_array` returns a numpy array like `.get_formatted_data` used to.
- The question "do we want a fixed-width output?", which is a formatting question, is translated like all formatting matters into a parameter `fixed_width` for the `GenericArrayFormatter` constructor (boolean, default `True`, because that's the main usage as of today)

This seems (to me) the simplest way to know which method should be used.

I have merged the two independent pieces of code that we used to have, and which were doing basically the same thing. But `FloatArrayFormatter.get_formated_data` was more refined, as it was using a mask to deal separately with the NaN values. I've kept that approach, but I don't know if it is the best option in term of performances.

All green

this looked ok last time. pls rebase and repush.

@jreback all set

does this actually change any features? e.g. implement the decimal fix? (or is that another PR), this just cleanup?

thanks!

No, that was just a cleanup, that's why I haven't touched the tests. Now I should be able to move on to the decimal parameter (issue #12031)
Improved documentation for DataFrame.join
State: closed

closes #12188 

I modified the description in DataFrame.join to make clear the difference with DataFrame.merge, also added examples.


Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
can you update

Done, sorry for the delay.

pls rebase/update

Sorry for the delay, I've been working to meet some deadlines for a project. I'll update in the next few days.

can you rebase/update

can you update according to comments

can you rebase / update?

I think the documentation is clear now. There are 3 examples, one using the dataframes original indexes and two joining using the key columns, the first one setting key as the index in both and the second one using on.

thanks @edublancas nice improvement!
WIP/ENH: allow categoricals in msgpack
State: closed

closes part of #7621

WIP, but wanted to put it up as is, since it probably interacts with  #12129
cc: @kawochen 

this is how I modified the schema - open to suggestions

``` python
# normal block
 {'dtype': dtype,
  'data': ExtType(0, <bytes>)}
# Categorical block
 {'dtype': 'categorical',
  'data': {
      'codes': {'dtype': 'int8', 'data': ExtType(0, <bytes>),
      'categories': {'dtype': dtype, 'data': ExtType(0, <bytes>)}
}
```


Labels:
name: Enhancement
name: Categorical
description: Categorical Data Type

Comments:
I have cherry picked this onto 0.18.0rc1, fixed the conflict and it worksforme™.

Can I submit a PR with it?

iirc last time this was looked at we were going for a slightly more general soln for extension types
I believe @kawochen made the changes in master to make this accessible

@chris-b1 
@kawochen 

It would be awesome if there was any chance of getting this into 0.18.0.

no but 0.18.1 won't be far away

@pwaller - feel free to submit a PR on my branch or just open up a new one if it's simpler.   

Looking at #12129, the Series constructor logic can be probably cleaned up, but I think the overall approach still needs to something like what I did here - extending the schema to store both categories and codes.

Submitted at https://github.com/pydata/pandas/pull/12573

closing in favor of #12573
DOC: reformat install.rst testing section
State: closed

TST: `run.test()` now defaults warnings based on release version


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Build
description: Library building on various platforms

Comments:
merged via https://github.com/pydata/pandas/commit/83fe3f3157b766eb497bdffd7ed3f57a3706859d
Iterableiterator
State: closed

If the approach is not overkill, it can be used also in `UTF8Recoder` and `UnicodeReader` from `pandas/common/io.py`. Closes #12153.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Internals
description: Related to non-user accessible pandas implementation
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
see #9496 

`TableIterator` in `io/pytables.py` is still a bit too cryptic to me, but I can try to take a look at it next week.

ok on doing `HDFStore` iterator in another issue.

I removed edits to `BotoFileLikeReader` because as far as I can judge the failing tests were due to it (through its ancestor `key.Key`) already being an iterable, and knowing nothing about this format/service it's hard for me to debug.

thanks @toobaz 

I have marked off `csv,stat,sas`, remaining are `hdf` & `msgpack` which are in #9496