BrandonFors/pandas-github-issues huggingface dataset:
ENC: better pandas typesetting in ipython nbconvert --to latex
State: closed

Currently, when IPython notebooks are converted to latex, pandas `DataFrame`s are appear as raw (verbatim) text. One simple solution is to include a `_repl_latex_` function. I have tried this solution and works. See https://github.com/ipython/ipython/issues/8261


Labels:
name: Enhancement
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
This is a nice idea!

Could you please add a basic test? Could be something as simple as `assert ds._repl_latex_() == ds.to_latex()`, just so we know this doesn't get broken in the future.

Also, this will need a release note in the what's new:
http://pandas-docs.github.io/pandas-docs-travis/contributing.html#documenting-your-code

Nice idea indeed, I was only wondering if there is a way to make this configurable? Maybe on the nbconvert side? 
As I can imagine you don't always want a full tex table, eg when writing a tutorial where you want to have it look like console input/output

Well, yes... I was thinking of a simple fix... but it would be nice to be able to pass options in the way that you can pass them no to `to_latex`. Maybe store those options in a singleton and reuse them? Is there any similar situation in pandas?

We already have `pd.set_options` and `pd.option_context`: http://pandas.pydata.org/pandas-docs/version/0.16.0/options.html

So that would be a pretty easy way to add this customization if you're interested. But (from my perspective), that shouldn't be a requirement for this PR.

@lmarti can you update with a test and options as described above

@jreback yes! in the next couple of days. Sorry for the delay.

@lmarti can you add a test

@lmarti can you add a basic tests for this?

@lmarti pls update according to the comments

@lmarti can you update according to comments

I have sent a new PR (https://github.com/pydata/pandas/pull/11399) that contains all the requested modifications. Thanks for the wait.
API: Sort keys for DataFrame.assign
State: closed

Closes #9777

Previously the order of new columns from `.assign` was arbitrary. For predictability, we'll sort before inserting.

We need to be comfortable with this change since we can't change behavior later with a keyword arg.

Technically we _could_ allow referencing the a column defined within the same call to `assign` as long as they are sorted. e.g. `df.assign(C=df.A + df.B, D=df.C**2)` would work, but not `df.assign(df.D=df.A +df.B, C=df.D**2)`. But I don't think we should.

cc @mrocklin


Labels:
name: API Design

Comments:
:+1:  This looks great to me!

agreed, you _may_ want to add an example in the docs where you show using _chained_ assignment (if for example you have an evaluation order requirement), .eg.

```
df = DataFrame({A = ..., B = ...})
df.assign(C=df.A+df.B).assign(D=df.C/df.A)
```

I've got one like that in the warning box

``` python
        (df.assign(C = lambda x: x['A'] + x['B'])
           .assign(D = lambda x: x['A'] + x['C']))
```

@TomAugspurger small not related note: you pushed your branch to upstream instead of master (no problem for now of course, but we should try to not do that for PRs I think)

@jorisvandenbossche are you talking about https://github.com/pydata/pandas/commit/f00d6bba5a7ee42569b9d6f7848962fd04a3360d ? I reorganize the whatsnew entry when merging that commit. Or I could be confused about what I did. But I think I pushed this branch to origin.

no, I was talking about this pr, you apparantly pushed it to upstream (see at the top of the PR, I noticed it because did a `fetch upstream`). But as I said, no problem, it can always happen! Just be sure to delete it after the pr is merged

@jorisvandenbossche I see what you mean now. Sorry about that. I'll clean it up when this gets merged.

Speaking of which, any objections to merging?

this looks ok, go ahead and merge
BUG: Repeated time-series plot causes memory leak
State: closed

Closes #9003, Closes #9307. 
- Not to clutter plot functions outside of classes.
- Changed to use classmethods rather than closures for plot funcs. 
- Not cache plot function itself on `ax._plot_data`. Changed to store `str` indicates plot kind.
- Moved `tsplot` related codes to `tseries.plotting.py`.

Because the change is little big, I think it is better to wait until 0.17. Further simplification seems to be possible if we can remove `tsplot` as not public function.

CC: @qwhelan, @TomAugspurger 


Labels:
name: Visualization
description: plotting
name: Performance
description: Memory or execution speed performance

Comments:
I moved this to 0.17.0, but can consider for 0.16.1 if warranted

@sinhrks Thanks for cleaning this up - looks good to me and I'd agree on waiting till 0.17

Being a n00b, not sure how to report this better: I think there is a misprint in the patch. In pandas/tseries/plotting.py line 45:

```
    freq, eries = _maybe_resample(series, ax, kwargs)
```

should be:

```
    freq, series = _maybe_resample(series, ax, kwargs)
```

Thank you for the amazing tool, and for keeping making it better!

@anton-molyboha Thanks for correction! Fixed this, and will add further tests by 0.17. 

Rebased, and added some tests for @qwhelan 's point. I think it's ready for review.

@sinhrks can you rebase / update?

OK, rebased.

@TomAugspurger @jreback Pls take a look when you have a time.

Just gave a read through. This is touching a lot of code, but I didn't see any big problems. Let's hope our tests are good enough. I'd say merge it when your ready.

Thanks for review! Merging.
TST: Split graphics_test to main and others
State: closed

Because `test_graphics.py` gets little longer, I've splitted it to 2 files. Test methods itself are not changed.
- `test_graphics.py` is for `.plot()` methods.
- `test_graphics_others.py` is for other plots, such as `.hist()`, `.boxplot` and miscs. 


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
OK, this seems fine by me... though we do eventually probably want to consolidate all these methods into `.plot.hist()` and so forth (I promise I'll finish that PR!)

@shoyer In plot accessor, hist and box calls `.plot(kind="hist")` and `.plot(kind="box")` internally. These tests are included in `test_graphics` because these are part of `.plot` method. So no worries.

Moved to "test_graphics_others.py" are  `DataFrame.hist` and `DataFrame.boxplot` which used prior to `.plot(kind="hist")`,  etc. These use separate (inconsistent) logic from `.plot`.

Is this OK after v0.16.1 visualization-related PRs are merged (#8885)?
@jreback, @TomAugspurger

Yeah. I have time set aside this weekend.  Will review then.

@sinhrks status?

@TomAugspurger  OK for this?

Yeah, the only thing I would maybe add is a module docstring at the top of the file explaining which tests go where. But go ahead and merge.

@TomAugspurger Thanks. Added short explanations to both files, and will merge after it gets green.
DOC: add dev environment creation details to contributing.rst
State: closed

- provides instructions for creating a development on any platform with conda
- cleans up the install.rst a bit


Labels:
name: Docs

Comments:
cc @rockg
@jorisvandenbossche 
@shoyer 
@cpcloud 

very nice, thanks!

I used a very similar procedure myself to setup my dev environment.

yep, this is one of the current pain points in getting started (esp on windows)

@jorisvandenbossche just wanted to get this in, if you have comments, ill do a follow up
BUG: format small floats correctly (GH9764)
State: closed

closes #9764 

In order to display a number from range (0,1) in decimal format with N significant digits (and not in scientific format), the number needs to be greater than or equal to 1e(-N+1), not 1e-N. 

Did not test for range of display.precision, as this does not seem to current practice.

Did not include tests as they would imply an output format which might change in the future.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
This should have a test to verify that your fix works. We use unit tests to verify our public API, but they're also for internal usage. The test will guarantee that nobody breaks this behavior without thinking carefully about it, even if we're not going to guarantee the output format to our users.

Sure, test added to branch.

looks pretty good -- could you add a line to the bug fix section of the release notes for v0.16.1?

also prob needs a test with mixed (meaning a really small number and a normal sized float). further this prob needs a quick way (eg vectorizes) to figure out whether to do this formatting (eg see how Timedelta does it for example) otherwise perf on a big array may be a problem

@shoyer Thanks for comments - updated. I have also noticed that some of my commit msgs were not 100% in line with the contribution recommendations, so I've updated them.

ok this lgtm

Thanks @tomazberisa!
BUG: bug in json lib when frame has length zero
State: closed

closes https://github.com/pydata/pandas/issues/9781

on master:

```
>>> df = DataFrame(columns=['jim', 'joe'])
>>> df
Empty DataFrame
Columns: [jim, joe]
Index: []
>>> df.to_json()
'{}'
```

mixed type will segfault:

```
>>> df['joe'] = df['joe'].astype('i8')
>>> df.to_json()
Segmentation fault (core dumped)
```

on branch:

```
>>> df.to_json()
'{"jim":{},"joe":{}}'
>>> df['joe'] = df['joe'].astype('i8')
>>> read_json(df.to_json())
Empty DataFrame
Columns: [jim, joe]
Index: []
```


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Awesome, thanks!

@shoyer i am not sure this actually closes https://github.com/pydata/pandas/issues/9781

this handles the case in [the comments](https://github.com/pydata/pandas/issues/9781#issuecomment-88884808) but the OP's issue may still persists.

OK, that may have been a bit premature :).

can you test your new code on that pickle file?

I hit this error in `0.16.0`
BUG: where behaves badly when dtype of self is datetime or timedelta, and dtype of other is not
State: closed

There are a few weird behaviors that this fixes:
1. If `other` is an `int` or `float`, then `where` throws an Exception while trying to call `other.view`
2. If `other` is an np.int64, it works fine
3. If `other` is an np.float64, there is no error, but the result is bizarre (it reinterprets the bits of the float as an integer, rather than casting it)
4. If `other` is list-like and has a numerical dtype, then it throws an exception while try to call `all` on the value False (for some reason, comparing an ndarray with dtype datetime64 and an ndarray with an integer dtype just returns a scalar False rather than a boolean ndarray)

```
>>> s = Series(date_range('20130102', periods=2))
>>> s.where(s.isnull(), 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/evanpw/Workspace/pandas/pandas/core/generic.py", line 3399, in where
    try_cast=try_cast)
  File "/home/evanpw/Workspace/pandas/pandas/core/internals.py", line 2469, in where
    return self.apply('where', **kwargs)
  File "/home/evanpw/Workspace/pandas/pandas/core/internals.py", line 2451, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/evanpw/Workspace/pandas/pandas/core/internals.py", line 1080, in where
    result = func(cond, values, other)
  File "/home/evanpw/Workspace/pandas/pandas/core/internals.py", line 1063, in func
    v, o = self._try_coerce_args(v, o)
  File "/home/evanpw/Workspace/pandas/pandas/core/internals.py", line 1819, in _try_coerce_args
    other = other.view('i8')
AttributeError: 'int' object has no attribute 'view'
>>> s.where(s.isnull(), np.int64(10))
0   1970-01-01 00:00:00.000000010
1   1970-01-01 00:00:00.000000010
dtype: datetime64[ns]
>>> s.where(s.isnull(), np.float64(10))
0   2116-06-17 06:38:37.588971520
1   2116-06-17 06:38:37.588971520
dtype: datetime64[ns]
>>> s.where(s.isnull(), [0, 1])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/evanpw/Workspace/pandas/pandas/core/generic.py", line 3326, in where
    if not (new_other == np.array(other)).all():
AttributeError: 'bool' object has no attribute 'all'
```


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
merged via https://github.com/pydata/pandas/commit/1f9b699b0d1f3bb41a2c7e92c15ac819bf601a72

thanks!
DOC: Fix broken formatting on docstring examples with first-line comments
State: closed

Examples for `interpolate` and `groupby` weren't being formatted correctly. See http://pandas.pydata.org/pandas-docs/version/0.16.0/generated/pandas.Series.interpolate.html and http://pandas.pydata.org/pandas-docs/version/0.16.0/generated/pandas.Series.groupby.html


Labels:
name: Docs

Comments:
Thanks for noticing! But, the 'correct' solution is that there should be a blank line between the explanation and the code. In this way the `code-block` is not needed, which looks a bit nicer in raw format.

Also the `#` is not needed then, so this can be removed as well.

see https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt#sections for a reference

But very welcome to update the PR!

@jorisvandenbossche ?

Yes, forgot about this. 
@evanpw Is looking good! Can you squash?

Squashed

Thanks!
API: define _constructor_expanddim for subclassing Series and DataFrame
State: closed

Closes #9762. Does this needs release note?


Labels:
name: Docs
name: API Design

Comments:
as an aside can u put a small doc that shows how to subclass DataFrame now that it is reasonably supportered - maybe put it in the FAQ

Sure, will do. By the way, is there any difference between FAQ and Gotcha? Otherwise, it may better to merge them to be more searchable.

@jorisvandenbossche what about merging faq and gotchas? I would say yes

Added subclassing documentation. Pls review.

@sinhrks This documentation on subclassing is great

I don't have much feedback on the particulars because I have not done much subclassing of dataframe myself.

However, we should clarify two things:
1. **Is this subclassing API officially supported?** My impression was that up until now, this has not been the case. We should think carefully about what we can guarantee this and in any case include a warning against using _other_ internal methods. Perhaps we should explicitly label this subclassing API as "experimental" (we reserve the right to break it in the future if necessary).
2. **We should encourage using composition instead of inheritance when possible.** This is a good opportunity to point out that the Index class, in particular, has a well defined API. For example of using composition with the pandas.Index to enable label based indexing, you could point users to [xray](http://github.com/xray/xray).

Nice docs! I agree with the questions raised by @shoyer, although I cannot say much about it myself as I never did some subclassing of dataframes.

But one remark: I don't think this belongs in a FAQ, but that we should start with some 'advanced' section in the docs (together with internals or so)

@sinhrks pls add a mention to try using `monkey-patching` first if you just want to add a _function_ http://pandas.pydata.org/pandas-docs/stable/faq.html#adding-features-to-your-pandas-installation
maybe put this as the first warning box! (2nd would be the composition/inheritance)

@sinhrks ok as @jorisvandenbossche just mentioned, let's move this to the internals.rst section.

Fixed @jreback 's points, and moved it to internals.rst. And preparing a composition example in separate PR.

@sinhrks lgtm, some minor comments.

I think you can add a release note in enhancements. Trying not to 'advertise' this too much :)

@jreback Thanks, fixed your points, and added a release note.

@sinhrks awesome.

I sent you an invite to change the pandas push permissions. As soon as you accept, pls make this your first PR to merged!

Thanks, let me merge this once after rebased.

@sinhrks awesome!

Did we ever resolve the question of whether the subclassing API is officially guaranteed or not? I suppose the reality is that people are going to subclass whether we tell them it's OK or not.

@shoyer well enough warnings to not encourage it
but u r right people will do it anyways

it's pretty doable nowadays (though imho composition is much more useful)
Closes #9795 (Stata writer changes input frame)
State: closed

closes #9795 

Just needed to move the data frame copy earlier in the execution path.


Labels:
name: Bug
name: IO Stata
description: read_stata, to_stata

Comments:
awesome!

pls add a release note in v0.16.1.txt

merged via https://github.com/pydata/pandas/commit/52a30d93a9d153ba1d8b30c430f6c49a6ce461c4

thanks!
DOC: regenerate CONTRIBUTING.md
State: closed

This is an alternative to #9796.

I simply did a couple of changes to the restructuredtext source so it generates cleaner markdown. The main thing was making the categories not quite so nested. (Was that [heavy nesting](http://pandas-docs.github.io/pandas-docs-travis/contributing.html#committing-your-code) intentional?) I think pandoc was running out of header styles or something like that.

cc @rockg @jreback @jorisvandenbossche


Labels:
name: Docs

Comments:
lgtm

What I had in mind for this actually just having a link to the pandas documentation in contributing.md.  The whole point of this was to have all the documentation in one place so they are never disconnected.

GitHub seems to want to have a CONTRUBUTING.md file to show to the user.
but I think we all agree it's better in the docs
not sure how we can have only a link (well we could but it's not as nice as a rendered markdown) in the CONTRIBUTING.md file 

I was thinking something like the below on the page:

Thank you for contributing to pandas.  Please see our guidelines [here](http://pandas-docs.github.io/pandas-docs-travis/contributing.html#contributing-to-the-code-base)

I also think having things not duplicated would be better. But I think it should be possible to 'include' the version in the docs into the CONTRIBUTING one with `..inlcude::`. Or does the CONTRIBUTING has to be markdown?

Otherwise I would also go for a link to the developers docs, but maybe with a kind of shorter 'summary'

Unfortunately a CONTRIBUTING.rst file doesn't work. So that rules out using an include.

I'm not opposed to making this file just a link, but it also seems safe enough to just generate it automatically with pandoc.... that seems like it will be straightforward enough to keep up to date.

I agree

let's just out the pandoc command in the wiki somewhere

ok!

Just put the instructions in the wiki, see the bottom here: https://github.com/pydata/pandas/wiki

To answer your question, the heavy nesting was intentional (and I still think we should find a way to do it in CONTRIBUTING).  Everything was setup to be sequential so after you checkout, then you either work on the docs or the code and then you commit.  So I think both the contributing to docs/code should be on same level as those two other things (and the rest of the workflow).

Maybe what we should be doing here is to pull out the `Contributing to the Code Base` into a separate .rst file and then only convert that subfile into contributing.md.  I think that is really the relevant section for ensuring a pull request is following the project's guidelines.  I can play around with that.

Oh, OK :). In that case, my mistake -- please feel free to revert. I'm guessing you might have mixed up subsection styles (e.g., `~~~~~~~` vs `""""""""""`) -- it looked like almost everything converted properly except for one section title.
DOC: add doc/source/contributing.rst as the CONTRIBUTING.md
State: closed

xref #9754 

I just did

`pandoc doc/source/contributing.rst -t markdown > CONTRIBUTING.md`

to generate the new CONTRIBUTING file.


Labels:
name: Docs

Comments:
cc @rockg
@jorisvandenbossche 

ok, so this is the question. I can edit this file, but then these are disconnected.

anyone used pandoc for this before?
BUG: DataFrame._slice doesnt retain metadata
State: closed

Closes #9776.


Labels:
name: API Design

Comments:
@sinhrks this is gr8.

I suspect there are some more places where this fails (e.g. look in `core/indexing.py`)

@jreback I looked through `generic.py` and `indexing.py`, and all the `_constructor` looks have required `__finalize__`, except for intermediate output purpose.

thanks!

interested in #8572 ?
ENH: Add StringMethods.partition and rpartition
State: closed

Derived from #9111.


Labels:
name: API Design
name: Strings
description: String extension data type and string data

Comments:
What do you think of having `return_type` defaults to 'frame'? (I know in `str.split` the default is 'series', but that is for back compatibility)

+1 for defaulting to return a frame.

OK, changed default to 'frame'.

lgtm, but not sure when would use `partition` over `split`

`partition` will always give three items back (only split on first or last occurence of separator) I think?

@sinhrks maybe add some examples to the docstring, and put a 'see also' to split explaining the difference. (+ a see also the the other (r)partition)

Added docstrings (examples and see also). `partition` is more specific than `split`, may useful when want to retain separators.

 @jorisvandenbossche Thanks, fixed your points.

About return type, I'll prepare separate PR for existing funcs which is also incorrect.

Hmm, `StringMethods` docstrings are mostly derived from corresponding module level functions, and there are 2 policies:
1. Describe the return type when called via `StringMethods` (describe what user will see)
   - https://github.com/pydata/pandas/blob/master/pandas/core/strings.py#L396
2. Describe the return type of the module level function (what the module level function returns)
   - https://github.com/pydata/pandas/blob/master/pandas/core/strings.py#L541

First policy looks preferable, so OK to change all the descriptions like first one? I understand that these module level functions are not public.

Yes, there is no real good solution to have it correct for both (unless starting with templating, but then the purpose of having a visual docstring you can correctly see in the source code for the module level functions is a bit lost). So I would say: most important that user sees the correct info, so I would indeed go with the first one.

@sinhrks or, maybe what we could also do is this: replace the array with Series, and leave the module level function docstrings as is. 
Eg for the example of `str_findall`: have a `.replace('matches : array', 'matches : Series')` on the docstring

BTW: it is not that there is a different policy in the docstrings apparently. Eg the `str_extract` you linked to _is_ really returning already a Series/DataFrame. So it is just that the `str_` functions have different return types.

Separate docstring issue to #9843 as we have to fix other methods, and also consider #9667.

@sinhrks can you update this with `expand=True/false` instead of return_type? (see discussion in  #9870)

OK, fixed this. And posting the behavior of `expand` in #9870.

looks fine. needs a rebase.

@jreback @jorisvandenbossche Thanks, fixed your points. Pls check again.

Looks good!
ENH: Add option in read_csv to infer compression type from filename
State: closed

Ideally, I would love for this to be the default, but that wouldn't be backwards-compatible in the case where the filename ends in '.gz' or '.bz2' and you want to treat it as uncompressed. That seems like it would be very rare, though.


Labels:
name: API Design
name: IO CSV
description: read_csv, to_csv

Comments:
I think it would even be fine even to change the default here. We are not _that_ strict about backwards compatibility in pandas -- any users who relied on the previous behavior were basically relying on a bug.

I agree with @shoyer here, let's just have it infer these filename endings as compression (move the release note to the API section).

I've totally borked this branch with an accidental force push. I'll fix it tonight.

Should be fixed.

couple of minor comments, pls rebase, and ping when green

Docs are fixed, rebased/squashed, and tests are green.

lgtm

@shoyer @jorisvandenbossche 

Looks great to me!

@evanpw thanks!

Thank you!