BrandonFors/pandas-github-issues huggingface dataset:
DOC: add dynamically generated doc-strings for generic functions
State: closed

closes #4717

```
In [2]: pd.DataFrame.reindex?
Type:       instancemethod
String Form:<unbound method DataFrame.reindex>
File:       /mnt/home/jreback/pandas/pandas/core/generic.py
Definition: pd.DataFrame.reindex(self, *args, **kwargs)
Docstring:
Conform DataFrame to new index with optional filling logic, placing
NA/NaN in locations having no value in the previous index. A new object
is produced unless the new index is equivalent to the current one and
copy=False

Parameters
----------
index : array-like, optional
    New labels / index to conform to. Preferably an Index object to
    avoid duplicating data
columns : array-like, optional
    New labels / index to conform to. Preferably an Index object to
    avoid duplicating data
method : {'backfill', 'bfill', 'pad', 'ffill', None}, default None
    Method to use for filling holes in reindexed DataFrame
    pad / ffill: propagate last valid observation forward to next valid
    backfill / bfill: use NEXT valid observation to fill gap
copy : boolean, default True
    Return a new object, even if the passed indexes are the same
level : int or name
    Broadcast across a level, matching Index values on the
    passed MultiIndex level
fill_value : scalar, default np.NaN
    Value to use for missing values. Defaults to NaN, but can be any
    "compatible" value
limit : int, default None
    Maximum size gap to forward or backward fill
takeable : boolean, default False
    treat the passed as positional values

Examples
--------
>>> df.reindex(index=[date1, date2, date3], columns=['A', 'B', 'C'])

Returns
-------
reindexed : DataFrame
```


Labels:

Comments:
easy enough

@jreback Everything that gets defined on NDFrame is available to all subclasses, right? So why bother specifying the classes?

its possible that a sub-has an implementtion already (but I guess I can just check that)

maybe will use some of this, most is now done in #5034

closing in favor of #5052
ENH: Add 'records' outtype to to_dict method.
State: closed

`to_json` has a records type that turns a DataFrame into a 

```
[{'a': 1, 'b': 3}, {'a': 2, 'b': 4}]
```

records json string.  It'd be nice if you could do a similar thing but not have to go `json.loads`.


Labels:

Comments:
grt....can you add a release notes entry (improvements), and a mention in v0.13.0? (use this PR ref as there is not associated issue)

Updated the commit - let me know of anything else.

@jtratner @cpcloud any comments?

@jtratner I updated the test to with your suggestion, seems to have done the trick.

ok....once more rebase and i think good 2 go (prob just release notes conflict)

thanks!

Thanks for the feature!

The resulting list of dicts has NumPy data types in entries, which is unfortunate -- difficult to serialize to JSON with known packages such as json.

Is there a chance to have an option to convert entry types to native Python for this?

Thanks
ENH/API: allow customization of requests thru opener
State: closed

closes #4927


Labels:

Comments:
@jreback @jtratner Is this _really_ necessary? @alefnula has a nice solution to the problem using `requests`: http://stackoverflow.com/a/18939272/564538.

can requests be added as an included library in pandas? 

it's still seeing active development ... i'm really hesitant to start using submodules

That seems like a better solution. So it becomes a dep for read_html?

Hold on, the SO solution is even simpler - if you want your own headers or anything else, pass in a file-like/data:

```
tables = pd.read_html(requests.get(url,
                                   headers={'User-agent': 'Mozilla/5.0'}).text,
                      attrs={"class":"tabelle_grafik"})[0]
```

Why write a lot of code to handle what other libraries can do easily for us?  We can't hope to cover every case anyways.

I think @hayd wanted to use it unstead of urllib in general

I guess not that big if a deal to add it

@jtratner Exactly my point ... users should just pass their own data in if necessary and wrap it in a try suite to fallback if a server can't handle some part of the request ... we can't handle _everything_

so just show in cookbook then! (or an example)

@cpcloud okay, then we should just close this and mark the other as wontfix - users should just do it themselves. Conversion of html --> NDFrame is in scope for pandas, getting data (outside of straightforward url), not so much.

yep. great. i'll add a cookbook entry
Scatterplot  Update for #3473
State: closed

Rebased Code for Scatterplot targetting bug #3473


Labels:

Comments:
@cpcloud 

Thanks for your extensive comments. I tried to address all of them although the choice of some of the defualts around subplotting I simply copied from another Class (Line or Bar , I forget which) - so I didn't really think through the behavior there.

Can I bug you about git one more time? To make my changed I made a new branch, `scatterplot3`,  and manually re-added my few commits. I can issue anew pull request from this branch or I can try to rebase this branch, `scatterplot2`,  to master and add my `scatterplot3` commits. What do you reccommend?

First things first:
1. [Hook up Travis-CI](https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md#steps-to-enable-travis-ci)
2. Write a smoke test in `pandas/tests/test_graphics.py` as a sanity check (I didn't see one).
3. If you don't yet have `upstream` as a remote then add it via `git remote add upstream git://github.com/pydata/pandas.git`

About `git`: does `scatterplot2` have any changes that you need? Or are they all in `scatterplot3`?

All the code is in `scatterplot3` except for the comments. I'll take a lookat some of the other graphics tests and make a new one for scatterplots. 

If all of your code is in `scatterplot3` then do this:

``` sh
git fetch upstream
git checkout scatterplot2
git reset upstream/master..  # reset everything back to, but not including, your current upstream/master
git stash  # just in case you need something that you forgot :)
git rebase upstream/master
git merge scatterplot3

# if all goes well
git stash drop stash@{0}
git branch -D scatterplot3
git push --force
```

Assuming you've set up your Travis-CI hook before doing all of that, it should subsequently run the test suite pretty soon after you run `git push --force`

@jorisvandenbossche Thanks for the code review, however, this branch should be closed. The updated code is in #3473. This PR should be closed/deleted. 

@zachcp OK!
BUG: Fix bound checking for Timestamp() with dt64 #4065
State: closed

closes #4065

To fix the bug, this change adds bounds checking to
_get_datetime64_nanos() for numpy datetimes that aren't already in [ns]
units.

Additionally, it updates _check_dts_bounds() to do the bound check just
based off the pandas_datetimestruct, by comparing to the minimum and
maximum valid pandas_datetimestructs for datetime64[ns].  It is simpler
and more accurate than the previous system.


Labels:

Comments:
can u test this in windows? should work but c code changes can sometimes do weird things 

Is there any easy way to do that online or do I need to dust off a windows VM?

http://pandas.pydata.org/pandas-docs/dev/install.html for 32-bit

64-bit a little bit non-trivial; there are some online guides
ENH: evaluate datetime ops in python with eval
State: closed

closes #4897

Also adds `DatetimeIndex` comparisons in query expressions and a corresponding vbench for a simple `Timestamp` vs either `Series` or `DatetimeIndex`.


Labels:

Comments:
maybe add some tests where use is comparing a date column with a non-date
? so should raise 

as an aside in future min/max/diff/shift on dates have to be done in python land

@jreback @jtratner comments?

this should eventually handle timedelta as well

under 1.7! they r like ints so I bet ne can handle them but prob better to handle in python because of NaT 

i ithink i'll do a new pr for that

:boom: 

```
In [5]: td = timedelta64(1, 'D')

In [6]: ne.evaluate('td + 1')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-f2c950d373ae> in <module>()
----> 1 ne.evaluate('td + 1')

/home/phillip/.virtualenvs/pandas/lib/python2.7/site-packages/numexpr/necompiler.pyc in evaluate(ex, local_dict, global_dict, out, order, casting, **kwargs)
    728
    729     # Create a signature
--> 730     signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)]
    731
    732     # Look up numexpr if possible.

/home/phillip/.virtualenvs/pandas/lib/python2.7/site-packages/numexpr/necompiler.pyc in getType(a)
    627     if kind == 'S':
    628         return bytes
--> 629     raise ValueError("unkown type %s" % a.dtype.name)
    630
    631

ValueError: unkown type timedelta64[D]
```

@jreback comments?

looks fine otherwise

moved to generic and added a smoke test for failing panel/panel4d with multiiindex

slight hit when using `DatetimeIndex` vs a column of `Series`, need to clear that up b4 merging

turns out it's because a `datetime64` `Series` with an `Int64Index` compares faster than a `datetime64` `Series` with a `DatetimeIndex`. This is the case before 8a9a4f2 (the just merged timestamp compare fix) so that didn't introduce any performance regressions (I also ran the full vbench suite on it and no changes). I ran a vbench on this as well and nothing changed.

bombs away

@cpcloud do you recall if we fixed this for timedelta ops as well (which should be evaluated in python space and not by numexpr)

Don't think there's a test for them in top level query/eval, but there are some tests in `test_pytables.py`.
DOC: show disallowed eval syntax
State: closed

closes #4898


Labels:

Comments:
@jreback @jtratner speak now or forever `git revert` your peace.

I'd say that this is not an exhaustive list.

e.g., "We've made an effort to keep this syntax simple and clean. As of now, these syntax elements are not supported and support is not planned for the future (not an exhaustive list)."

> I'd say that this is not an exhaustive list.

I did look at the set of disallowed nodes for the `pandas` parser, so I would say it is (unless I missed something), as far as user-facing syntactic categories go.

great - then never mind :)

Here are the disallowed expression nodes:

```
frozenset(['Set', 'SetComp', 'Repr', 'Yield', 'Call', 'DictComp', 'Dict', 'IfExp', 'GeneratorExp', 'Lambda'])
```

So you're right, I missed yield and generators

I think `Repr` is for interactive stuff....let me check

maybe have allowed syntax just to make it obvious?

I mean in addition to disalllowed

let me see how long that list is ....

list is pretty long .. i'll just add a smaller list encompassing a bunch of ops per bullet point

@jreback @jtratner merge?

I would switch order, put allowed first, maybe call it `Allowed syntax?` - or if you can think of a better name

okay ... what do u think?

good
CLN: more plotting test cleanups
State: closed

Just a couple of cleanups here and there

Adds a new decorator to skip matplotlib in a class method


Labels:

Comments:
@jreback merge? just a couple of clean ups regarding testing and added a decorator to inject a matplotlib test skip for `unittest.TestCase` classes.

this looked ok 

FYI, seeing `ImportError: No module named nose` in clean virtualenvs (w/o nose) after the util/testing.py change in this PR was merged

yes ... better to just remove debug from `pandas.__init__.py`

and i'll move `nose` out of the toplevel import
BUG/TST: Allow generators in DataFrame.from_records
State: closed

closes #4910
- nrows implementation doesn't allow unknown size iterator like
  generators, if nrows = none ends with a TypeError.
- To allow generators if nrows=None consume it into a list.
- Add two tests of generators input.


Labels:

Comments:
pls add release notes (`doc/source/release.rst`), referencing the issue
pls hook up travis (see contributing.md)

thanks!

Sorry for the mess, I'm new to github. 
I've tested it locally with python 2.7 and 3.3 but didn't work with 2.6. Now it's all ok.

I recognize that this may be silly, but how is this supposed to work if you passed something like this:

``` python
gen = ('a' for _ in range(50))
DataFrame.from_records(gen)
```

Just wondering if this needs to be caught anywhere or if this is okay

I suppouse it's ok, the patch only put the generator in a list in the same fashion as main constructor.
DataFrame.from_records() expects a collection of records, which every record it's a collection of fields: list, tuple, dict, or in your case a string wich it's a sequence (collection) of characters (only one character in your example).

it's fine, it looks like it just works the same as a list. Doesn't need to be added as a test case either.

@tinproject can you rebase and squash to commit? otherwise looks ok

Hope it's ok, if it's not please give me a git command example, as i said i'm new to git :(

pretty good instructions here: https://github.com/pydata/pandas/wiki/Using-Git, let us know 

I think it's now ok. Release notes file release.rst conflicts but are trivial and I thought I solved it.

try `git rebase -i upstream/master` need to remove that merge of master commit

@jreback now it's all ok, no? I must improve on git, sorry.

yes...looks good.....let me review

git just comes from practice! submit more PR's!

@tinproject thank you!
Fixing issue #4902.
State: closed

closes #4902, Added a check for complex numbers.


Labels:

Comments:
@alefnula Can you add a test if you aren't doing that already? Thanks :)

release notes as well!

@alefnula just fyi those are in `doc/source/release.rst`

Uh sorry, I'm not that familiar with the pandas project workflow yet. Can someone point me what should I add to release notes, and in which test suite/test file should the test for `pandas.core.nanops._ensure_numeric` go? The function is also private for the module and I don't see it used anywhere else, so should I test it directly or using just the public api.

I'm really sorry for this many questions. :-/

@alefnula No problem.
1. As I said above, release notes are in `pandas/doc/source/release.rst`
2. Test it on something similar to the SO question, e.g., create a heterogeneously typed `DataFrame` and compute the mean. Something like,

``` python
def test_complex_mean(self):
    df = DataFrame({'a': tm.choice(['a', 'b'], size=10), 'b': rand(10).astype(complex) + randn(10) * 1j})
    result = df.mean().item()
    expected = df['b'].values.mean()
    self.assertEqual(result, expected)
```

@cpcloud OK got it. One more question. `test_series.py` - is this an appropriate place for this test? Since this happens on every aggregate operation on a Series object.

@alefnula yes...can put it there....try to find a similar group of tests and try it (note that make sure its NOT under SafeForSparse), because I don't think this works on sparse

@jreback I just found a `test_ensure_int32` int `test_common.py`. Maybe this is a better place for `test_ensure_numeric` since it's testing the exact change?

@alefnula that's fine too

@alefnula do you have travis hookup up? see contributing.md for instructions...

Added tests, updated the `release.rst`.

@jreback Enabled travis. But saw your post too late, already pushed the changes. :-/

go ahead and set up travis, then

`git commit --amend -C HEAD`
will trigger travis with a new commit hash

@jreback Succeeded :) Than you and @cpcloud for assistance. :)

looks good....pls rebase and squash to 1 commit and should be mergeable

no need to close ....

@cpcloud I'm haven't closed it. Github automatically closed it when I rebased and pushed.

strange

yeah, that's not supposed to happen at all. very weird.

wonder if the commit message has naything to do with it?

It happened two times... When I did the amend and when I rebased :-/

that's very weird since "fixing" is not [one of the words that closes an issue](https://help.github.com/articles/closing-issues-via-commit-messages)

thanks!
BUG/TST: Fix failing FRED comparison tests and remove skips.
State: closed

Replacing my mess of a PR in #4890.

Fixes #4827

Assertions were being made about arrays (of one item) equaling
a scaler.


Labels:

Comments:
just so you are aware...if you push a new branch with the same name as your PR branch, that's what Github uses. So if you ran :

``` sh
# delete fixfred3 remotely
git push origin :fixfred3
git branch fixfred3 master
git checkout fixfred3
git push -u origin fixfred3
```

Then Github would pick up the change and use it for the PR. No magic involved. (though now it won't work because it's already been merged).
BUG/TST: Fix failing FRED comparison tests.
State: closed

Fixes #4827

Assertions were being made about arrays (of one item) equaling
a scaler.


Labels:

Comments:
thanks :) Can you rebase on master and remove the skip tests?

Oh hell, I messed up git.  Give me a minute to figure this out :) 

@jtratner I'm not sure how to dig myself out of this hole. I figured it would be easier to start anew at #4891.
CLN: move README.rst to markdown
State: closed

Because GitHub's rst is ugly


Labels:

Comments:
sexy :)

FYI, one of the reasons GitHub's rst is ugly, is not the fault of GitHub, but because the rst in README.rst was not wel formatted .. 

Lists should not be indented (something also in the whatsnew and release notes is done wrong), that's why they looked like 'quotes' in the rendering of the README.

hm i seem to remember shoddy support for rst even with correct, and very basic markup

If I look at the README.rst, I think the lists is the only issue, apart form not nice rendering of internal links as  `:func:`~pandas.io.html.read_html`` (but you converted them to normal links in the markdown version).

But maybe it has improved over time.

maybe. do you not like the new version? i think it's simpler and more clear

does this mess up how PyPI looks?

arg ... yes it will .... could convert markdown to rst and put that in long
description but that's annoying ... have to change back
https://wiki.python.org/moin/CheeseShopTutorial

## 

Best,
Phillip Cloud

On Mon, Sep 23, 2013 at 6:31 PM, Jeff Tratner notifications@github.comwrote:

> does this mess up how PyPI looks?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4888#issuecomment-24960865
> .

@cpcloud No, I don't really bother about md or rst in the readme. I just saw the issue because there was someone complaining about because of the broken build.  
I think the only thing I find less clear in the md version are the headers if you look at the plain text.

And I don't think it is a problem for PyPI, it is already using the LONG_DESCRIPTION in setup.py (but also there, the indentation is wrong).
Panel tshift 4853
State: closed

closes https://github.com/pydata/pandas/issues/4853

I think I'm doing this right. I moved the `tshift` into generic and had `shift` offload to `tshift` if `freq` is found. 


Labels:

Comments:
err, let me get Travis CI running on my fork. it passes nosetests on my local.

- travis (I see you already started that!)
- release notes

quick q, for the commits that make up a PR, do they need to follow the ENH/BUG/etc format? Or does everything get smushed into a single commit anyways when merged into master. 

if u r going to squash then try to make the top one be descriptive
i try to make them all though

You'll want to squash your commits down to a single commit before it's merged.

```
git fetch upstream
git checkout panel-tshift-4853
git rebase -i upstream/master
```

This should open your editor (`-i` stands for interactive) and you can follow the instructions there to squash or fixup your commits to just one.  Looking at your commits, you could probably squash the second into the first and then choose fixup for the other two.

Once that's done you can use `git push origin panel-tshift-4853 -f` to push to the branch in this PR.  Be careful with the `-f` option though.  This will mess things up if others are working off your branch.

Does this change Series, DataFrame or any sparse object behavior? (given
that you're changing NDFrame) If so, should test.

@dalejung I did a PR on your branch, https://github.com/dalejung/pandas/pull/1 which removes `shift` from series and dataframe and puts it in generic.py you can cherry-pick this.

@jreback cherry picked the cleanup, added release notes, and enabled travis

great looks good
will review soon

err, sorry hold off a bit. I started generalizing the shift infrastructure so Panel could use it. Need to split off into its own branch

okay, split off the additional work into its own branch/issue.

@dalejung let me know if you want to merge this (or finish the other PR first)

@jreback This can be merged first. thx

alright..looks fine...can you squash down a bit?

squashed. sorry used to hg where the commits don't carry over after merge. This PR has definitely leveled up my git-fu.

thanks!
BUG: Constrain date parsing from strings a little bit more #4601
State: closed

closes #4601

Currently dateutil will parse almost any string into a datetime.  This
change adds a filter in front of dateutil that will prevent it from
parsing certain strings that don't look like datetimes:

1) Strings that parse to float values that are less than 1000
2) Certain special one character strings (this was already in there,
   this just moves that code)

Additionally, this filters out datetimes that are out of range for the
datetime64[ns] type.  Currently any out-of-range datetimes will just
overflow and be mapped to some random time within the bounds of
datetime64[ns].


Labels:

Comments:
Yeah that looks like a much better place.  Will do.

what does this do with an out-of-bounds date? raise? (or depends on coerce)?

i believe this should close #4065 as well?

@danbirken I realize there is a `pandas.core.datetools` (which is not that big)...should prob go there

It treats an out of bounds date as an invalid date string, so it will either coerce it or not convert the array to datetimes.  It doesn't throw an error.

As for the location, both of those other places were causing some circular dependency issues, and on second thought I don't think either of them are any improvement over just putting them in tslib.pyx (which is a file full of timestamp related functions anyway).  I also think the tests being in test_tslib.py is an improvement over the previous change.

#4065 is similar, but this change wont fix it.  This only fixes the specific case of parsing timestamps from strings (like csv parsing) -- previously out-of-bounds timestamps would just be mapped randomly within the appropriate bounds, whereas now they will be seen as invalid.  I think fixing that one should be pretty easy though, I can do that after this one.

gr8 on both points

Rebasing so it can be merged cleanly.  Let me know if there is anything else I should add to get this merged.

can make tr function names a little shorter

maybe call the dunction parse_datetime_string?

and I think u can add kw argument, maybe validate to parse_datetime_string to call the verify bounds function  (make this function name shorter too)

Shortened the function names and added the kw param.

Can you make sure all your test function names + open parens are under 80 characters in line length? 

If you feel the test needs a longer explanation, you can put it in comments below the function name. 

Sorry I am used to 100 column style rules, my bad.  Fixed everything to pep8 80 cols.

totally reasonable - 100 character makes sense too, we just use 80 everywhere. Thanks for making the change! I'll leave it to @jreback for deciding to merge this.

@danbirken thanks for the fixes!