BUG: scatter_matrix problem, ranges of diagonal and off-diagonal plots are different
State: closed

close #5497

**Short story:**

In [this](https://github.com/pydata/pandas/issues/5497) issue I signaled that the x_lim and y_lim in scatter_matrix plots are different. In particular x/y_lim of diagonal hists and x/y_lim of scatter plots are different.  

For instance, this causes peaks nonalignments in the 1-margin and 2-margin of the plotted multivariate pdf. See pictures in the issue.

This is a possible fix.

**Long Story:**
In a `scatter_matrix` plot we have a diagonal constituted by histograms and `scatter_plots` to show correlation among variables in the subplots away from the diagonal.

Let's consider this scatter matrix in which we have two variables `A` and `misfit`
![pict](https://f.cloud.github.com/assets/2300692/1522061/389753b0-4ba1-11e3-97f0-31caab31fcb7.png)

The ranges of `A` and of `misfit` are shown by the axis thicks along the boundary subplots (i.e. plots in the first row and column (starting form the bottom left corner)).

As we can see there is a clear misalignment of the peak among plot 1,2 and plot 2,2. 
This is due to the fact that the plot range for each plot (`xlim`) is determined automatically by matplotlib, hence there is a possible disagreement among different plots. 

In this pull request, the `xlim`s and `ylim`s are forced to be consistent and are calculated as the real ranges of the variables (max - min) plus a given correction (here 5%).  

i.e., for each plot

```
   rdelta_ext = (rmax_ - rmin_)*xy_range_extension  # xy_range_extension defaults 5%
   ax.set_xlim( (rmin_ - rdelta_ext , rmax_+ rdelta_ext)  )
```

This is an example of a `scatter_matrix` obtained with the correction (please compare e.g subplots 1,2 and 2,2 )

![pe_model_radial_const_force](https://f.cloud.github.com/assets/2300692/1782665/5a1afe04-68b3-11e3-89df-51f3884eff91.png)


Labels:

Comments:
@TomAugspurger, would you mind reviewing this for 0.13.1?  

@y-p sure  thing. Today or tomorrow. 

@acorbe I haven't had a close look yet, but is the idea of this PR to provide a kwarg so the user can play with it till it looks good? Were you able to figure out why they weren't aligned in the first place?

@TomAugspurger 

hi. The idea is the following.

Originally, the plot ranges (`xlim` and `ylim`) of the different subplots are determined by the matplotlib internal routines. 

In particular, `plot` determines the range (`xlim`) for the diagonal plots, while `scatter` determines the range for the plot out of the diagonal. What happens is that such ranges are different one another (by some internal matplotlib mechanics, I guess) and this causes the wrong alignment of the first picture above.

In this PR, the `xlim`s of the subplots in the columns (and the `ylim`s of the subplots in the rows) are forced to be the same. As a consequence the plots are correctly aligned.

On the other hand, a value for the `xlim`s should be properly set. This is chosen according to the following.

Let `values` be the first column of the `DataFrame`. Its pdf will be associated to the first row and column of the `scatter_matrix`. The `xlim` of the subplot in the first column will be 

```
    rmin_ , rmax_ = min(values),max(values)
    rdelta_ext = (rmax_ - rmin_)*xy_range_extension
    finalrangechosen = (rmin_ - rdelta_ext , rmax_+ rdelta_ext)
```

i.e. the data range `[minvalue, maxvalue]` is enlarged in percentage by a value 2*`xy_range_extension`. 
Such value is defaulted at 5% but can be changed by the user. This allows one to have a small nice blank space around the plots - i.e. the plots don't touch the subplots frames [better for scatter plots, imho].

In other words, the plots will always be aligned; besides, the user has freedom to choose the size of the small blank frame around each subplot (which is defaulted to 5% of the function range).

What do you think?

Thanks for the explanation, that sounds great.

Changes to the code:
1. Should `xy_range_extension` take a percent (x in [0, 100]) or a decimal (x in [0, 1])? I guess this is like the `alpha` kwarg which expects numbers in [0, 1](which is the current behavior of this PR). Maybe change the docstring to explain that `xy_range_extension` is converted to a percentage, or just say that the default is .05 to give a hint to the user.
2. `scatter_matrix` doesn't follow the [numpy docstring format](https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt#sections), but you may as well change `xy_range_extension` to follow the format there. `param: type`, newline, then explanation. (You can update the other parameters if you want!)
3. My in-line comments.

A couple more things, and then this should be good to go:
1. release notes entry in `doc/source/release.rst` with a line explaining the problem. I guess this could go under either the API or Bug Fixes section, probably bug fixes.
2. A test in `pandas/tests/test_graphics.py`. Model it after the other ones in there. We just want to make sure we have something that hits the new keyword, and you may as well check that the `lim`s are being set correctly.

(Edited to save y-p's and everyone's eyes) Here's a before and after: Looks much better!

Before:

![before](https://f.cloud.github.com/assets/1312546/1950047/56914f4a-8131-11e3-984d-64bccfcfcfc7.png)

After:

![after_5](https://f.cloud.github.com/assets/1312546/1950048/5b6ff0c0-8131-11e3-9c4d-2210b7f02cac.png)

You'll need to rebase off the current master also. Holler if you run into any git problems.

oh no, evil blue again. Please use `pd.options.display.mpl_style='default'`.

Thanks @TomAugspurger, looks good to me.

@acorbe, if you can fix up the mentioned issues I'll merge this for 0.13.1, Thank you.

@y-p Sure thing. Give me an evening or two.

@TomAugspurger. Thanks for the review! I'll fix the things. I'll try to work out the rebase on my own. I'll write you in case I fail.

Dear @TomAugspurger and @y-p ,

I guess the job is done with the rebasing and the new function doc according to numpy docstring format.

A sample from the latest compiled code 

```
import pandas as pd
import numpy as np

a = np.random.normal(size = 1000)
b = np.exp(a)

c = np.array([a,b]).transpose()

df = pd.DataFrame(c)

from pandas.tools.plotting import scatter_matrix

pd.options.display.mpl_style='default'

scatter_matrix(df)
```

![figure_1](https://f.cloud.github.com/assets/2300692/1962439/673863fe-8287-11e3-8ded-8f4e0342afa2.png)

What do you think?

Please give me feedback

Best regards,
Alessandro

Looks good to me, see note on proper way to add a new keyword.

@jreback objections to merging for 0.13.1? this was marked 0.14.0 previously.

Finally, we should do at least a little bikeshedding on the keyword name as it's
a bit of a mouthful now. Anyone familiar with terms used for similar functionality
elsewhere? `feather` perhaps? `fudge`?

@y-p 

you're right for the keyword; I'll fix its position once we decide the good one.

Can it be some `relative_frame_size`, `frame_size`, `framesize`?

Unfortunately, I am not the guy to find smartly fitting words in English.

Let's borrow from the css box-model and call it `range_padding`.

ok for 0.13.1

Keyword has been substituted with `range_padding` in code, documentation and test.
Position has been corrected.

Squashed and merged as 2f24ff2

Thanks a lot for this @aterrel,  we appreciate it. 

There are lots of open graphics bugs issues for pandas and they don't get much love. 
We'd be glad to have more such bug fixes from you if you're so inclined.

Thanks for the PR @acorbe.

I have some more bug fixes. I'll push them.

It's more a matter of necessity than inclination..