Repository: statsmodels/statsmodels

Issue:
is sm.open_help ever defined?
Near the end of `statsmodels.api`:

```
chmpath = os.path.join(os.path.dirname(__file__), 'statsmodelsdoc.chm')
if os.path.exists(chmpath):
    def open_help(chmpath=chmpath):
        from subprocess import Popen

        p = Popen(chmpath, shell=True)
```

I'm not familiar with "statsmodelsdoc.chm".  Under what conditions does that exist?


Discussion:
Used to be there for windows installs.  It is a MS-specific **c**ompiled **h**t**m**l format. 
> Used to be there for windows installs.

Is it still useful?
Probably not.  I don't think it ever gets produced. 
I haven't looked for it in years.
It was very convenient when I got started and I had python, numpy, scipy and statsmodels chm open most of the time. At that time I maintained the statsmodels chm file.
Now I'm reading documentation only occasionally and then I either use the source in Eclipse or use google search and a browser which is faster for occasional use and which also provides a link for quoting.

I still prefer local chm-like documentation when I need to refer to it more often, e.g in Stata. I found it annoying when R dropped the built-in doc viewer and replaced it by opening a browser.


> I haven't looked for it in years. [...] I still prefer local chm-like documentation

To clarify, is this still useful?  It sounds like the function never actually gets defined, in which case this piece of code's main function is to confuse people like me.
I don't know where anyone would get the chm to use. Is it every built?
It was not automatically built.
I had build it directly using the sphinx option with a different css theme than the web docs and uploaded it "somewhere" and/or included in the source distribution for Windows, according to https://pypi.org/project/scikits.statsmodels/ Section Windows Help.
https://launchpad.net/statsmodels/+download
(first few links in google search for statsmodels.chm)

A bit similar, for a while we also tried to maintain the pdf file of the docs but there were always problems with the chapter and section hierarchy that sphinx created.
I was a big fan of chm files, I even build for a while my own chm for pandas and for numpy, scipy after those two dropped the official chm files.
Just opened #5168 that removes this.

Is there dev documentation on the doc build/release process?  I think that might be helpful for figuring out which files in tools/ and possibly doc/ are actually used.  Also just Bus Factor diligence.

So... remove code that only serves to cause confusion?  #5168.
So... wait for a contributor that builds htmlhelp and similar doc files?

Python still distributes chm files which I still use https://www.python.org/downloads/windows/

(I might even try it myself again when I update my python setup)

Yep, wait and then reintroduce the code if and only if it becomes useful.  Until then, recognize that many readers are not you and un-documented dead code is counterproductive.
It should be clear to you by now that I do NOT agree to stripping down statsmodels to code that is currently fully supported.
"Carthago delenda est" doesn't work with me. 
I'm still on a 10-year plan, and we are not close to the statsmodels 1.0.0 house cleaning.
E.g. A lot of my "infrastructure work" in statsmodels is not immediately useful and is not immediately used. But it provides the foundation to add new features much faster in future. It is only undocumented dead weight if we would freeze statsmodels at status-quo features.

E.g. `python -c 'import statsmodels.api as sm; sm.show_versions();'` just got resurrected also as a CI function and not just as a user function.
(It wouldn't exist anymore if I had listened to contributors that seem to be happy deleting a third of statsmodels.)

(Also there were some items in the early development that Skipper removed and I didn't want to fight for it. But then I had to start from scratch to reimplement it several years later.
It's the trade-off between cluttering the API and code and making code and API future proof or resistant.)

e.g. future proofing API, function signature
```
method=None
   currently only xxx is available but additional methods will be added in future. The default might or will change, so use the keyword to keep a specific method
```

"Status-quo is just a moment in time, the past and the future are much longer."
> "Carthago delenda est" doesn't work with me.

Do... do you think that deleting code can't be undone?  Any analogies with Salting The Earth are inappropriate.

> A lot of my "infrastructure work" in statsmodels is not immediately useful and is not immediately used.

Yep, and there's a place in statsmodels for code fitting this description.  Granted, a better place would be "outside of master", and it isn't nearly isolated enough, but sandbox is the place such code, not api.

> It is only undocumented dead weight if we would freeze statsmodels at status-quo features.

Again you seem to be forgetting that many readers are not, in fact, you.  For most non-you readers, non-functional non-documented code is precisely dead weight.

> But it provides the foundation to add new features much faster in future. 

You know what else helps with that?  Teamwork.
"Carthago delenda est" 
I checked the "modern usage" afterwards, based on English and German Wikipedia version.
English usage seems to put emphasis on "warfare", i.e. about Carthago
German usage refers to "wenn eine Forderung beharrlich wiederholt wird.",
translation by Google: "when a claim is persistently repeated." i.e. to Cato's speeches.

I meant the German usage of ending every speech with the same request/demand/claim, and didn't want to imply anything about warfare.

I don't think that a few lines of openhelp function are really confusing. We have another one like that somewhere which is for html, which I also haven't used in years.
Readers can just skip over it.

Aside: I do have another 30,000 to 50,000 lines "outside of master" 
https://github.com/statsmodels/statsmodels/pulls/josef-pkt?page=2&q=is%3Aopen+is%3Apr+author%3Ajosef-pkt
but that has the problem that I/we/contributors cannot do incremental enhancements and improvements, and consequently 80 to 90% finished code lingers for many years.

> Again you seem to be forgetting that many readers are not, in fact, you. For most non-you readers, non-functional non-documented code is precisely dead weight.

I don't think you or anyone is required to read statsmodels in it's entirety. Many things, like code comments are much more understandable based on the specific context, history and planned development.
There doesn't seem to be a problem to making context specific contributions like #4721.
The fact remains that I'm a main developer and the main general reviewer and I need to be able read the code including the "small print", and help others like GSOC students to get through the code.

I will stick to that "no cleanup when in doubt" policy for the near future, and I don't want to have the same comments and same explanation in every second PR, which I wanted to point out with the Cato quote.










> translation by Google: "when a claim is persistently repeated."

I'm kind of surprised that it would translate as anything other than "Carthage must be destroyed." 
 Heh, silly me, thinking that words mean things.  

> I don't think that a few lines of openhelp function are really confusing.

Like many other things, this is one of many very small issues that add up across the codebase.

> Readers can just skip over it.

How are readers supposed to know what to ignore?

> The fact remains that I'm a main developer and the main general reviewer and I need to be able read the code including the "small print"

Refusing to make the code accessable to non-you readers perpetuates this cycle.  This is not conducive to a healthy community.
This thread has left me wondering whether statsmodels is trying to be a community project or if it is just a grouped set of individual contributors code where the person who wrote the code "owns" it (or their delegate if they have left).

It is not helpful to claim that there at XXk lines of code out there ready to go and so nothing (or little) is allowed to go in that would conflict with these PRs written 2+ years ago.  If something is ready, then it should go in.  If it isn't ready, and you want it in, get it ready.  If you don't get it ready then it looks like you don't really want it in; and if you don't want it in, then let someone else work on improving that piece of code. 

If you don't want much of statsmodels to be a community project then you should make this clear; failing to do this leads to frustration and ultimately loss of potential contributors. As @rgommers pointed out, contributing guidelines would be a good place to start.
statsmodels was never a community project in the sense of a large permantent core maintainer base. Until Skipper left, it was he and I that wrote most of the code outside of GSOC projects and we were the only ones discussing the general direction.

statsmodels was always a community project in the sense that there were and are many contributors and many users providing feedback in issues.

Since Skipper left I am the only one planning and "monologuizing" about the general direction and structure of statsmodels. However, there are areas like GLM where there were many contributor, especially thequackdaddy that contributed both to the design and to the improvements.
Kerby and Chad contributed groups of large models that they maintain and are busy with those areas  seem to be only marginally interested in other parts. So they are the main maintainers in their areas.
You, Kevin, and thequackdaddy have essentially taken over CI and build issue, so I only need to keep an eye out for serious side effects but I am not trying to understand most details anymore.

Being a community project doesn't mean that software development is based on a democracy. We need long term maintainers that have code "ownership" and who feel responsible for the continued development and the direction an area is going. Changes to statespace models have to go through Chad, changes to mixed model and others have to go through Kerby. Most of the other things currently have to go through me. This also means that in general I feel responsible for substantive reviewing and helping out with any unclear issues or issues how something fits in statsmodels and can reuse current features.

In contrast to this, I don't see a big priority of across the board style and code cleanup, and I especially don't like it when it affects "my" working code, history and future plans, or conflicts with my maintenance tasks.

About the XXk lines of code: I managed to get several things from 2015 in because I didn't have a GSOC student this summer. I have a NUMFocus sponsored/finance project to get two more in. Instead I have been flooded for a month with cleanup PRs. And I don't want to delegate merging cleanup PRs when they affect "my" code.

To repeat it again: we need a lot of context specific review and refactoring but not cleanups for their own sake jumping all across the code base.

example GLM which has many issues and already many merged PRs during the last years to improve it
https://github.com/statsmodels/statsmodels/issues/2753
https://github.com/statsmodels/statsmodels/issues/2804
https://github.com/statsmodels/statsmodels/issues/1720
https://github.com/statsmodels/statsmodels/issues/4759
and many other especially to fix scale handling for the general case.

This is in my opinion a good example for where the community is at work in a community project.
In contrast to this, deleting some extra stray comments or fixing whitespace all over the place, or deleting a currently unused function, is low priority, and might conflict with content development.

> let someone else work on improving that piece of code.

There are several that are free for the taking, e.g. Skipper's tobit PR needs review and redoing all the Stata results so they are not 200,000 LOC.
BetaRegression needs updating and derivatives. heckman needs decision whether it is two in one or should be one or the other model.
GAM is what I signed up for.
(I'm not so sure about some of my PRs, e.g. robust PRs are several weeks of work, treatment effects, IPW and similar are still a design mess in my PR even though it has already most of the methods that Stata has.)
but if nobody volunteers, then those stay on my todo list.

I also advertised for extending diagnostics to WLS and GLM but haven't seen a volunteer yet.








I've tried to stay out of these "steering committee" discussions because I really don't have a dog in the fight. I'm just a user. But I just had a very nice dinner with beer so I'm a little relaxed and restless, so I'll give a few thoughts.

1. Whatever the steering committee decides, I think we should avoid fragmentation, forks, mass opening/closing of PRs and the like. One of my favorite Steve Jobs rants was an investor conference call where he defended Apple's iOS walled garden ecosystem against the Google Android free-for-all approach suggesting that it makes life easier for users and developers. In a way, statsmodels is very similar. If we go the R route where every little function is its own package with its own maintainer, I think you get into nasty problems where each tool can do one thing nicely, but none can do all. statsmodels genmod (IMHO) is far better for my use case because we have fairly good formula language support (although I still find patsy slow), you can lazily load your data, we have all the EDM families, the scipy optimizers are fast, and the diagnostics/summary screen is very intuitive. And since this is python, I can do the data munging I need to do in pandas and make models in pretty notebooks. Re-creating all that in R/spark is not worth my time. It took a long time for me to get happy with it, but I mostly am. 
2. statsmodels is already in a very competitive space. R has tools that basically does everything statsmodels can do (albeit more clumsy/slow/annoying IMHO). stata/SAS/SPSS/Revolution R/whatever are generally bug-free (albeit not price-free), Julia is supposed to be that hot girl who just transferred to your school, spark does "big data". There's already lots of tools, and we need to find a way to market this coherently without confusion for potential users. 
3. statistics is taught in college, but I personally spent too much time playing quake 3, half life, and drinking and honestly didn't understand it nearly as well as I do now... and I still don't understand most of the math here. God help me if I ever do. I think I'm relatively smart--but not on the same level as some of the folks here--but of pretty much everyone I interact with in my day job, I only know of 1 or 2 other people who could contribute a feature-enhancement PR (outside of spellchecking documents). The skillset is rare... decent at programming, moderate-to-advanced understanding of statistics, and needing functionality above introductory college-level statistics. 
4. @josef-pkt is the main technical support contact. Just today there was an issue which (IMHO) had a relatively obvious answer, that being the user saw an unexpected degrees of freedom in his model because his exog was not full rank. Despite that, no one (including myself) spoke up to ask the user "hey that feature is pretty strongly unit tested and real world tested now, did you try checking that your data is full rank." He's the only person that answers questions regularly on stack overflow. With the exception of some time series questions, he's likely the only person that responds on the mailing list. The fact that he's the only person with the time/energy/patience/knowledge should speak to the fact that statsmodels covers some very hard terrain. Alternatively, I think if some of the other people here began trying to take on these technical support issues, it might free the maintainer to take up more design and code-review and less answering on stack overflow
5. I'm a little conflicted on some of @jbrockmendel 's PEP8 crusade. Certainly, statsmodels is of such an age that PEP8 wasn't a thing thing. Lady Gaga had just released Pokerface and Slumdog Millionaire won Best Picture. Code was written, it (mostly) worked, and in the years since only got refactored when there was a bug. I definitely think re-factoring working code simply to make it PEP8 compliant in and of itself is a distraction. But I do see merit in fixes of bug-wrong items. If a model/diagnostic returns an incorrect parameter/diagnostic, I think fixing it--along with PEP8 cleanup--should be a fast develop/test/review/merge process. 
6. I've swayed back and forth on suggesting we modular-ize statsmodels. Clearly, Chad Fulton seems to understand time series and he should be (and I believe basically is) empowered to be the BDFL pro tem for TSA-related PRs. Kevin Sheppard up-to-speed with the python packaging/documenting/unit testing/deploying. I think appointing him BDFL pro tem on that domain PRs make sense. I'm missing other contributors from this list not out of disrespect, but just what I've noticed from posts/PRs. However, going completely modular--with a separate lead for each section--is going to miss some big picture overlap that we need. Lots of diagnostics/techniques are very recyclable from topic to topic (see optimization, ANOVA, summary, etc.) and you don't want to let every maintainer do his own thing. You'll never get a coherent tool (see Steve Jobs rant above). But I do think there might be some middle ground.
7. There likely should be some succession planning. This jives with a random project at my day job, but there is a large risk to this project if one or more of Josef/Chad/Kevin/Kerby loses interest or becomes suddenly incapacitated (god forbid). This isn't as easy as appointing someone by a vote. Clearly, they have to have the energy, focus, skill set, employment flexibility, and family understanding to to respond to tens of messages per day about myriad topics which can be obscure. I know the people here mainly by what they've commented/submitted, and this is exceedingly difficult to do for years on end. How Josef does it, I have no idea. 
8. I think there's a decent usecase for statsmodels in many analytics jobs, but we don't really have a bench of evangelists who go to conferences to educate potential users or encourage people to download/try/break/engage/enhance statsmodels. It would be nice if we could get more publicity by getting us on the agenda at pycons. I think the actuarial community, for one, would love to see some the things we can do. Part of that maybe would be a closer relationship with people like anaconda/enthought who can help businesses download and use the tool. 
9. In a google hangout with some close friends, I frequently use salty language to get a laugh. Especially when a [certain](https://www.youtube.com/watch?v=FfkVcV9nlOI) [sports](https://youtu.be/McaBtB0qwys?t=119) [team](https://youtu.be/ONDRuzyZKt8?t=486) [embarrasses](https://youtu.be/9OsHFzG8NWU?t=170) [themselves](https://www.sbnation.com/lookit/2014/9/13/6145893/texas-forgets-how-coinflips-work-gives-ucla-ball-to-start-both-halves), I love haranguing my roommate. But its very dangerous to use acerbic language here. I've never met anyone here and what might seem fun and harmless in person can be misinterpreted by the reader. I think we should be a little dull in the language we use. 

Anyways, the alcohol has worn off and Jets v Browns is over. Good night.
@jbrockmendel comment deleted

I'm not going to take this much longer without heavier censoring.

And I'm not stepping down like a BDFL who got tired of ...
Should be removed.

Resolving patch:
diff --git a/statsmodels/api.py b/statsmodels/api.py
--- a/statsmodels/api.py
+++ b/statsmodels/api.py
@@ -54,18 +54,6 @@
 
 load = load_pickle
 
-chmpath = os.path.join(os.path.dirname(__file__), 'statsmodelsdoc.chm')
-if os.path.exists(chmpath):
-    # As of 0.10.0, this is not reached.  See GH#5134
-
-    def open_help(chmpath=chmpath):
-        from subprocess import Popen
-
-        p = Popen(chmpath, shell=True)
-
-del os
-del chmpath
-
 from ._version import get_versions
 __version__ = get_versions()['version']
 del get_versions


Test patch: