Repository: pre-commit/pre-commit-hooks

Issue:
Non-utf8 file will cause trailing-whitespace to truncate the file contents
Hi,

(I started investigating this in https://github.com/pre-commit/pre-commit/issues/397)

Bug reproductions steps, after setting up a git repository with the "trailing-whitespace" pre-commit hook installed (with the latest sha: 775a7906cd6004bec2401a85ed16d5b3540d2f94):
```
echo -en '\x3c\x61\x3e\xe9\x20\x0a\x3c\x2f\x61\x3e' > minimal_crasher.xml
git add minimal_crasher.xml
pre-commit run trailing-whitespace --files minimal_crasher.xml
```

Results in:
````
Trim Trailing Whitespace...............................................................................................................................................................Failed
hookid: trailing-whitespace

Files were modified by this hook. Additional output:

Fixing minimal_crasher.xml
Traceback (most recent call last):
  File "/home/lucas_cimon/.pre-commit/repok3wjweh4/py_env-default/bin/trailing-whitespace-fixer", line 9, in <module>
    load_entry_point('pre-commit-hooks==0.6.0', 'console_scripts', 'trailing-whitespace-fixer')()
  File "/home/lucas_cimon/.pre-commit/repok3wjweh4/py_env-default/lib/python3.4/site-packages/pre_commit_hooks/trailing_whitespace_fixer.py", line 71, in fix_trailing_whitespace
    _fix_file(bad_whitespace_file, all_markdown or extension in md_exts)
  File "/home/lucas_cimon/.pre-commit/repok3wjweh4/py_env-default/lib/python3.4/site-packages/pre_commit_hooks/trailing_whitespace_fixer.py", line 12, in _fix_file
    for line in fileinput.input([filename], inplace=True):
  File "/usr/lib/python3.4/fileinput.py", line 263, in __next__
    line = self.readline()
  File "/usr/lib/python3.4/fileinput.py", line 363, in readline
    self._buffer = self._file.readlines(self._bufsize)
  File "/home/lucas_cimon/.pre-commit/repok3wjweh4/py_env-default/lib/python3.4/codecs.py", line 319, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte
```

And **all the content of `minimal_crasher.xml` lost** !!

In this test the xml file is very small, but it can happen with any file size nor number of staged modifications !


Discussion:
asottile: Needs a test, also I'm not the biggest fan of fileinput, I think it might be easier just using `io.open` instead?  (that'd remove the awkward temporary file that may clobber other files in the working directory).
Lucas-C: Test added
Lucas-C: I'll do the move to `io.open` in a 2nd commit, when implementing the "option 2" we discussed.
asottile: New test seems falling :(
coveralls: 
[![Coverage Status](https://coveralls.io/builds/7522605/badge)](https://coveralls.io/builds/7522605)

Coverage remained the same at 100.0% when pulling **ab1afc7680c9210bb1782e2cce1ebf814068d341 on Lucas-C:master** into **775a7906cd6004bec2401a85ed16d5b3540d2f94 on pre-commit:master**.

coveralls: 
[![Coverage Status](https://coveralls.io/builds/7522605/badge)](https://coveralls.io/builds/7522605)

Coverage remained the same at 100.0% when pulling **ab1afc7680c9210bb1782e2cce1ebf814068d341 on Lucas-C:master** into **775a7906cd6004bec2401a85ed16d5b3540d2f94 on pre-commit:master**.

coveralls: 
[![Coverage Status](https://coveralls.io/builds/7524557/badge)](https://coveralls.io/builds/7524557)

Coverage remained the same at 100.0% when pulling **211beb11487534c0462201ad1fc6964ed92a0a8b on Lucas-C:master** into **775a7906cd6004bec2401a85ed16d5b3540d2f94 on pre-commit:master**.

coveralls: 
[![Coverage Status](https://coveralls.io/builds/7525502/badge)](https://coveralls.io/builds/7525502)

Coverage remained the same at 100.0% when pulling **3cf15fdbfec70f379265a2152852249909af6daa on Lucas-C:master** into **775a7906cd6004bec2401a85ed16d5b3540d2f94 on pre-commit:master**.

coveralls: 
[![Coverage Status](https://coveralls.io/builds/7525502/badge)](https://coveralls.io/builds/7525502)

Coverage remained the same at 100.0% when pulling **3cf15fdbfec70f379265a2152852249909af6daa on Lucas-C:master** into **775a7906cd6004bec2401a85ed16d5b3540d2f94 on pre-commit:master**.

coveralls: 
[![Coverage Status](https://coveralls.io/builds/7525795/badge)](https://coveralls.io/builds/7525795)

Coverage remained the same at 100.0% when pulling **ad38d9bfc490e34409415eba305e1cbd3f21757f on Lucas-C:master** into **775a7906cd6004bec2401a85ed16d5b3540d2f94 on pre-commit:master**.


Patch:
diff --git a/.travis.yml b/.travis.yml
index 41f9e7d..de7c294 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,9 +1,8 @@
 language: python
-python: 3.5
 env: # These should match the tox env list
     - TOXENV=py27
+    - TOXENV=py33
     - TOXENV=py34
-    - TOXENV=py35
     - TOXENV=pypy
     - TOXENV=pypy3
 install: pip install coveralls tox
diff --git a/README.md b/README.md
index ee77b25..6c3a3ec 100644
--- a/README.md
+++ b/README.md
@@ -52,9 +52,8 @@ Add this to your `.pre-commit-config.yaml`
     - Use `args: ['--django']` to match `test*.py` instead.
 - `pyflakes` - Run pyflakes on your python files.
 - `pretty-format-json` - Checks that all your JSON files are pretty
-    - `--autofix` - automatically format json files
-    - `--no-sort-keys` - when autofixing, retain the original key ordering (instead of sorting the keys)
-    - `--indent ...` - Control the indentation (either a number for a number of spaces or a string of whitespace).
+    - Use `args: ['--autofix']` to automatically fixing the encountered not-pretty-formatted files and
+    `args: ['--no-sort-keys']` to disable the sort on the keys.
 - `requirements-txt-fixer` - Sorts entries in requirements.txt
 - `trailing-whitespace` - Trims trailing whitespace.
     - Markdown linebreak trailing spaces preserved for `.md` and`.markdown`;
diff --git a/appveyor.yml b/appveyor.yml
index 30eab46..cf8f927 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -1,10 +1,10 @@
 environment:
     matrix:
         - TOXENV: py27
-        - TOXENV: py35
+        - TOXENV: py34
 
 install:
-    - "SET PATH=C:\\Python35;C:\\Python35\\Scripts;%PATH%"
+    - "SET PATH=C:\\Python34;C:\\Python34\\Scripts;%PATH%"
     - pip install tox
 
 # Not a C# project
diff --git a/pre_commit_hooks/detect_aws_credentials.py b/pre_commit_hooks/detect_aws_credentials.py
index 4c51546..f727078 100644
--- a/pre_commit_hooks/detect_aws_credentials.py
+++ b/pre_commit_hooks/detect_aws_credentials.py
@@ -4,7 +4,7 @@ from __future__ import unicode_literals
 import argparse
 import os
 
-from six.moves import configparser
+from six.moves import configparser  # pylint: disable=import-error
 
 
 def get_your_keys(credentials_file):
diff --git a/pre_commit_hooks/trailing_whitespace_fixer.py b/pre_commit_hooks/trailing_whitespace_fixer.py
index c159071..fa9b7dd 100644
--- a/pre_commit_hooks/trailing_whitespace_fixer.py
+++ b/pre_commit_hooks/trailing_whitespace_fixer.py
@@ -1,24 +1,27 @@
 from __future__ import print_function
 
 import argparse
-import fileinput
 import os
 import sys
 
 from pre_commit_hooks.util import cmd_output
 
 
-def _fix_file(filename, markdown=False):
-    for line in fileinput.input([filename], inplace=True):
-        # preserve trailing two-space for non-blank lines in markdown files
-        if markdown and (not line.isspace()) and (line.endswith("  \n")):
-            line = line.rstrip(' \n')
-            # only preserve if there are no trailing tabs or unusual whitespace
-            if not line[-1].isspace():
-                print(line + "  ")
-                continue
+def _fix_file(filename, is_markdown):
+    with open(filename, mode='rb') as file_processed:
+        lines = file_processed.readlines()
+    lines = [_process_line(line, is_markdown) for line in lines]
+    with open(filename, mode='wb') as file_processed:
+        for line in lines:
+            file_processed.write(line)
 
-        print(line.rstrip())
+
+def _process_line(line, is_markdown):
+    # preserve trailing two-space for non-blank lines in markdown files
+    eol = b'\r\n' if line[-2:] == b'\r\n' else b'\n'
+    if is_markdown and (not line.isspace()) and line.endswith(b'  ' + eol):
+        return line.rstrip() + b'  ' + eol
+    return line.rstrip() + eol
 
 
 def fix_trailing_whitespace(argv=None):
@@ -64,14 +67,13 @@ def fix_trailing_whitespace(argv=None):
                 .format(ext)
             )
 
-    if bad_whitespace_files:
-        for bad_whitespace_file in bad_whitespace_files:
-            print('Fixing {0}'.format(bad_whitespace_file))
-            _, extension = os.path.splitext(bad_whitespace_file.lower())
-            _fix_file(bad_whitespace_file, all_markdown or extension in md_exts)
-        return 1
-    else:
-        return 0
+    return_code = 0
+    for bad_whitespace_file in bad_whitespace_files:
+        print('Fixing {0}'.format(bad_whitespace_file))
+        _, extension = os.path.splitext(bad_whitespace_file.lower())
+        _fix_file(bad_whitespace_file, all_markdown or extension in md_exts)
+        return_code = 1
+    return return_code
 
 
 if __name__ == '__main__':
diff --git a/pylintrc b/pylintrc
new file mode 100644
index 0000000..c905a37
--- /dev/null
+++ b/pylintrc
@@ -0,0 +1,22 @@
+[MESSAGES CONTROL]
+disable=bad-open-mode,invalid-name,missing-docstring,redefined-outer-name,star-args,locally-disabled,locally-enabled
+
+[REPORTS]
+output-format=colorized
+reports=no
+
+[BASIC]
+const-rgx=(([A-Za-z_][A-Za-z0-9_]*)|(__.*__))$
+
+[FORMAT]
+max-line-length=131
+
+[TYPECHECK]
+ignored-classes=pytest
+
+[DESIGN]
+min-public-methods=0
+
+[SIMILARITIES]
+min-similarity-lines=5
+ignore-imports=yes
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 2922ef5..97343d5 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,7 +1,9 @@
 -e .
 
+astroid<1.3.3
 coverage
 flake8
 mock
 pre-commit
+pylint<1.4
 pytest
diff --git a/setup.py b/setup.py
index f92f821..e8c5000 100644
--- a/setup.py
+++ b/setup.py
@@ -11,19 +11,20 @@ setup(
     author='Anthony Sottile',
     author_email='asottile@umich.edu',
 
+    platforms='linux',
     classifiers=[
         'License :: OSI Approved :: MIT License',
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
-        'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: Implementation :: CPython',
         'Programming Language :: Python :: Implementation :: PyPy',
     ],
 
-    packages=find_packages(exclude=('tests*', 'testing*')),
+    packages=find_packages('.', exclude=('tests*', 'testing*')),
     install_requires=[
         # quickfix to prevent pep8 conflicts
         'flake8!=2.5.3',
diff --git a/tox.ini b/tox.ini
index f2ac125..a9b17c0 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,7 +1,7 @@
 [tox]
 project = pre_commit_hooks
 # These should match the travis env list
-envlist = py27,py34,py35,pypy,pypy3
+envlist = py27,py33,py34,pypy,pypy3
 
 [testenv]
 deps = -rrequirements-dev.txt
@@ -12,6 +12,7 @@ commands =
     coverage report --show-missing --fail-under 100
     pre-commit install -f --install-hooks
     pre-commit run --all-files
+    pylint {[tox]project} testing tests setup.py
 
 [testenv:venv]
 envdir = venv-{[tox]project}


Test patch:
diff --git a/tests/check_merge_conflict_test.py b/tests/check_merge_conflict_test.py
index f1528b2..8141ade 100644
--- a/tests/check_merge_conflict_test.py
+++ b/tests/check_merge_conflict_test.py
@@ -12,6 +12,9 @@ from testing.util import get_resource_path
 from testing.util import write_file
 
 
+# pylint:disable=unused-argument
+
+
 @pytest.yield_fixture
 def f1_is_a_conflict_file(tmpdir):
     # Make a merge conflict
diff --git a/tests/trailing_whitespace_fixer_test.py b/tests/trailing_whitespace_fixer_test.py
index 6f4fdfd..3a72ccb 100644
--- a/tests/trailing_whitespace_fixer_test.py
+++ b/tests/trailing_whitespace_fixer_test.py
@@ -42,7 +42,7 @@ def test_fixes_trailing_markdown_whitespace(filename, input_s, output, tmpdir):
 MD_TESTS_2 = (
     ('foo.txt', 'foo  \nbar \n  \n', 'foo  \nbar\n\n'),
     ('bar.Markdown', 'bar   \nbaz\t\n\t\n', 'bar  \nbaz\n\n'),
-    ('bar.MD', 'bar   \nbaz\t   \n\t\n', 'bar  \nbaz\n\n'),
+    ('bar.MD', 'bar   \nbaz\t   \n\t\n', 'bar  \nbaz  \n\n'),
     ('.txt', 'baz   \nquux  \t\n\t\n', 'baz\nquux\n\n'),
     ('txt', 'foo   \nbaz \n\t\n', 'foo\nbaz\n\n'),
 )
@@ -103,3 +103,12 @@ def test_no_markdown_linebreak_ext_opt(filename, input_s, output, tmpdir):
 
 def test_returns_zero_for_no_changes():
     assert fix_trailing_whitespace([__file__]) == 0
+
+
+def test_preserve_non_utf8_file(tmpdir):
+    non_utf8_bytes_content = b'<a>\xe9 \n</a>\n'
+    path = tmpdir.join('file.txt')
+    path.write_binary(non_utf8_bytes_content)
+    ret = fix_trailing_whitespace([path.strpath])
+    assert ret == 1
+    assert path.size() == (len(non_utf8_bytes_content) - 1)