GitHub PullRequestReviewCommentEvent at 2014-03-31 16:51:33
Repository: qmcs.github.io
Actor: marianomora
Payload: {"comment":{"url":"https://api.github.com/repos/qmcs/qmcs.github.io/pulls/comments/11124776","id":11124776,"diff_hunk":"@@ -0,0 +1,187 @@\n+======================================\n+Software in your reproducible research\n+======================================\n+\n+:date: 2014-03-28 17:22\n+:tags: cw14, programming\n+:category: workshop\n+:author: Dmitrijs Milajevs\n+\n+.. A big note to reviewers!\n+\n+    Please write you comments in this pull request\n+    https://github.com/qmcs/qmcs.github.io/pull/19\n+\n+    In case you want to contribute, create pull request to this fork (branch cw14)\n+    https://github.com/dimazest/qmcs.github.io/tree/cw14\n+\n+`Collaborations Workshop 2014 (CW14) <http://software.ac.uk/cw14>`__ took place\n+in Oxford in the end of March. I was lucky to attend the last day of the meeting\n+dedicated to ad-hock hacking. I would characterize the main topic of the\n+workshop as introduction of qualitative software development in scientific\n+environment.  Here are my thoughts, why `reproducibility is a dream that easily\n+becomes a nightmare`__.\n+\n+__ https://twitter.com/dimazest/status/449484586717048832\n+\n+-- PELICAN_END_SUMMARY --\n+\n+I'll start by comparing scientists to professional programmers and operations\n+and argue that for an experiment to be reproducible the software has to be\n+reusable. Then, I'll give some suggestions on how to make a reproducable setup.\n+\n+Scientists are developers\n+=========================\n+\n+`Mark Basham <http://pydata.org/ldn2014/speakers/#168>`__ at `PyData London\n+<http://pydata.org/ldn2014>`__ stated, what I consider, the pain of researchers\n+in one sentence. Here is my free recall:\n+\n+    Nowadays, scientists have to process so much data that they **have to**\n+    become programmers.\n+\n+I completely agree with it. It's not possible anymore to successfully run an\n+experiment without using highly optimized libraries for computation, IO and\n+result representation.\n+\n+I would go even further and explain the growing popularity of Python and other\n+high level languages by the fact that they hide complex implementation behind\n+simple interfaces (see `scikit-learn's`__ `cross validation code`__ for an\n+example).\n+\n+__ http://scikit-learn.org/stable/\n+__ https://github.com/scikit-learn/scikit-learn/blob/6b38d3b3051b4be298d4df4978dc35b56d5eb3a6/sklearn/cross_validation.py\n+\n+Therefore, readability and understandability of code plays an important role. To\n+the extend that when the code is being written it's very tempting to use the\n+development version of a library. `IPython 2`__ together with `Python 3.4`__,\n+anyone (bu the moment of writing IPython 2 is still being developed and Python\n+3.4 has been recently released)?\n+\n+__ https://github.com/ipython/ipython/blob/0485089180ff70feac77bd01bf23a410a787d8e5/docs/source/whatsnew/development.rst\n+__ http://docs.python.org/dev/whatsnew/3.4.html\n+\n+In this mode a scientist behaves like a developer. The more recent is the\n+software the better.\n+\n+Scientists are not operations\n+=============================\n+\n+In a software company, eagerness of developers is usually compensated by wisdom\n+of operations, the people responsible for running and supporting software.\n+\n+operations:\n+    /ɒpəˈreɪʃ(ə)ns/\n+\n+    `noun`\n+\n+    1. people who are responsible for deploying and monitoring\n+       services in a company.\n+    2. the action of functioning or the fact of being active or in effect\n+\n+In academia, usually, there is much less interest to code. Not everyone is\n+interested to study how an experiment is implemented. The main deliverable from\n+a scientist-developer is a result, whcih usually is just a number, that\n+hopefully beats the current state of the art. In addition, scientists-developers\n+have a strong opinion on what tools to use and prefer to reinvent a wheel,\n+rather reuse code writen by other scientists.\n+\n+The problem is that it's not easy to reuse code. At least it has to be\n+documented and well written. Both of the points require a lot of effort, which\n+could not be appreciated.\n+\n+.. Now it's a bit messy\n+\n+Scientists should be operations\n+===============================\n+\n+Why should scientist care about quality of their code? The main reason is that\n+eventually good code will be reused by someone else, and, finally, safe time.\n+Someone else could be yet another person in a group, or in a group in another\n+research center. Also, having a widely adopted tool minimizes amount of\n+surprises. For example `NLTK's BNC reader`__, which takes care of `some corner\n+cases`__. (A note for a careful reader: the amount of `regexp based code written\n+to process XML`__ is at least 10 times larger than you think.)\n+\n+__ https://github.com/nltk/nltk/blob/develop/nltk/corpus/reader/bnc.py\n+__ https://github.com/nltk/nltk/issues/70\n+__ https://twitter.com/dimazest/status/442723017958129664\n+\n+The bad news is that even `developers are still working on easy code reuse`__, aka\n+packaging, to make operations' live easier!\n+\n+__ http://maurits.vanrees.org/weblog/archive/2013/05/holger-krekel-re-inventing-python-packaging-testing\n+\n+Since scientists are already developers, they should learn from operations. This\n+means that the environment the code runs should be similar from the moment it's\n+being developed to the moment it's being deployed. This prevents bugs and\n+minimizes works-for-me situations.\n+\n+A good project\n+==============\n+\n+The documentation of a good project should include testing and deployment\n+instructions.\n+\n+The process should be automated as much as possible. `Tox\n+<http://www.testrun.org/tox/>`__ is a tool that provides a unified interface to\n+run tests and hides differences across different test runners.\n+\n+In the Python word, usage of eggs and listing of the requirements in\n+``setpup.py`` and their versions in ``requirements.txt`` make deployment\n+easier, but solves only half of the problem: it takes care only of Python\n+dependencies.\n+\n+`Vagrant <http://vagrantup.com>`__ is a great tool for virtual machine\n+management. It allows to use a completely isolated environment as if it was\n+local. For example, on my Mac I can start a vagrant box with CentOS 6.5 and have\n+the same environment as the computing server I'll use.\n+\n+Taking this into account, this should be necessary steps to run/develop\n+scientific software:\n+\n+.. code-block:: bash\n+\n+    # Get the Vagrant config\n+    git/hg clone http://project-site.org/project\n+    cd project\n+\n+    # Start the virtual environment\n+    vagrant up\n+\n+    # Connect to it\n+    vagrant ssh\n+\n+    # All system dependencies are installed\n+    cat README.rst\n+\n+.. code-block:: rst\n+\n+    ====================\n+    Dialogue act tagging\n+    ====================\n+\n+    The environment to run experiments described in Joe Doe. 2015. The Ultimate Dialogue act tagging.\n+\n+    Refer to http://project-site.org/ for more information.\n+\n+    The experiment data is stored ``data/``. To test the setup run:\n+\n+        tox\n+\n+    To run the experiment run:\n+\n+        bin/tagger doe2015\n+\n+.. code-block:: bash\n+\n+    # Now I know what to do and happily run the experiment\n+    bin/tagger doe2015\n+    Tagging accuracy is 100%.\n+\n+All this fancy and clear setup requires loads of love and care. A virtual\n+machine image has to be built and hosted somewhere. It has to be updated time to","path":"content/articles/013-cw14.rst","position":183,"original_position":183,"commit_id":"103843ba7df7d42780ece741907ef44edb554258","original_commit_id":"103843ba7df7d42780ece741907ef44edb554258","user":{"login":"marianomora","id":7117791,"avatar_url":"https://avatars.githubusercontent.com/u/7117791?","gravatar_id":"b0760cfdaac26399ba383707342c6db1","url":"https://api.github.com/users/marianomora","html_url":"https://github.com/marianomora","followers_url":"https://api.github.com/users/marianomora/followers","following_url":"https://api.github.com/users/marianomora/following{/other_user}","gists_url":"https://api.github.com/users/marianomora/gists{/gist_id}","starred_url":"https://api.github.com/users/marianomora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marianomora/subscriptions","organizations_url":"https://api.github.com/users/marianomora/orgs","repos_url":"https://api.github.com/users/marianomora/repos","events_url":"https://api.github.com/users/marianomora/events{/privacy}","received_events_url":"https://api.github.com/users/marianomora/received_events","type":"User","site_admin":false},"body":"\"from time to time\"","created_at":"2014-03-31T16:51:33Z","updated_at":"2014-03-31T16:51:33Z","html_url":"https://github.com/qmcs/qmcs.github.io/pull/19#discussion_r11124776","pull_request_url":"https://api.github.com/repos/qmcs/qmcs.github.io/pulls/19","_links":{"self":{"href":"https://api.github.com/repos/qmcs/qmcs.github.io/pulls/comments/11124776"},"html":{"href":"https://github.com/qmcs/qmcs.github.io/pull/19#discussion_r11124776"},"pull_request":{"href":"https://api.github.com/repos/qmcs/qmcs.github.io/pulls/19"}}}}