Test for Python 3.5 with C locale
State: closed

As @jreback suggested, I'm adding an alternate py3 build to change LOCALE, based on the 3.4 slow build, to reveal some encoding bugs (see #12337)

I'm new to configuring Travis, I just hope this will work as I expect and that the tests will fail.
- [ ] closes #12337 
- [x] tests added but not passed?
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Unicode
description: Unicode strings
name: IO LaTeX
description: to_latex

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/14114?src=pr) is 85.24% (diff: 100%)

> Merging [#14114](https://codecov.io/gh/pydata/pandas/pull/14114?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #14114   diff @@
==========================================
  Files           140        140          
  Lines         50556      50556          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43095      43095          
  Misses         7461       7461          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [289cd6d...dadf73c](https://codecov.io/gh/pydata/pandas/compare/289cd6d0df66b812921ff4c5cbade937b875406d...dadf73c0222fe7f49d6d0e179cc54dd37d642a14?src=pr)

This is disappointing. I added a travis job with LC_ALL=C, and the tests still succeed: https://travis-ci.org/pydata/pandas/jobs/155766390

The new locale seems to be taken into account, since travis says

```
$ ci/script.sh
inside ci/script.sh
Setting LC_ALL to C
/home/travis/build/pydata/pandas/pandas/computation/__init__.py:19: UserWarning: The installed version of numexpr 2.4.4 is not supported in pandas and will be not be used
  UserWarning)
pandas detected console encoding: ANSI_X3.4-1968
```

But on my computer,

```
$ LC_ALL=C nosetests pandas.tests.formats.test_format                                                     
..................S..............................................SSS.........E........................................................................................................
======================================================================
ERROR: test_to_latex_filename (pandas.tests.formats.test_format.TestDataFrameFormatting)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nicolas/Documents/Projects/pandas/pandas/tests/formats/test_format.py", line 2808, in test_to_latex_filename
    df.to_latex(path)
  File "/home/nicolas/Documents/Projects/pandas/pandas/core/frame.py", line 1655, in to_latex
    encoding=encoding)
  File "/home/nicolas/Documents/Projects/pandas/pandas/formats/format.py", line 661, in to_latex
    latex_renderer.write_result(f)
  File "/home/nicolas/Documents/Projects/pandas/pandas/formats/format.py", line 897, in write_result
    buf.write(' & '.join(crow))
UnicodeEncodeError: 'ascii' codec can't encode character '\xdf' in position 7: ordinal not in range(128)

----------------------------------------------------------------------
Ran 182 tests in 3.860s

FAILED (SKIP=4, errors=1)
```

I'm missing something, obviously. I'm going to try with Python 3.5, which is what I'm using on my computer, but I don't see how that could change things.

so the JOB_NAME determines which files to pick up in the `ci\requirements-` dir.  You need to add a copy of the files for the job which you copied. For something like this we would only need minimal dependencies (so its faster). you can zonk most of them (e.g. I assume you copied `requirements-3.5.run` (and `.pip` and `.build`), so edit the `.run` and remove evetything but `pytz,dateutil,numpy`

There is no `requirements-3.5.pip` . And I guess you meant `JOB_TAG` determines the files to pickup in the `ci/` folder?

I don't understand, the new test does not show up in Travis. It was the case with the previous commit, even though the new test was not in the "allowed failures". But it disappeared with my last commit...

@nbonnotte You have to add it in both places (both in the base matrix, as repeat it in the allowed failures section. As you had it in the first commit I think)

It works! I mean, it fails: https://travis-ci.org/pydata/pandas/jobs/157552565

great!

I think something up with the locale string itself - it doesn't seem to fully parse (error above the latex one); the final error is tagged in another issue 

```
======================================================================
ERROR: test_set_locale (pandas.tools.tests.test_util.TestLocaleUtils)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tools/tests/test_util.py", line 72, in test_set_locale
    lang, enc = LOCALE_OVERRIDE.split('.')
ValueError: not enough values to unpack (expected 2, got 1)
```

this looks wrong

@jreback I've set the locale to "C", which unlike "en_US.UTF-8" or "fr_FR.UTF-8" doesn't have any dot in it. Is it wrong? How should I set the system encoding to ascii?

Edit: I'll try with ANSI_X3.4-1968

@nbonnotte I have no idea what to put here

`- LOCALE_OVERRIDE="zh_CN.UTF-8`

looks like you need, country.encoding maybe? look thru python docs

http://stackoverflow.com/questions/19709026/how-can-i-list-all-available-windows-locales-in-python-console maybe

ANSI_X3.4-1968 doesn't work:

```
  File "/home/travis/build/pydata/pandas/pandas/__init__.py", line 4, in <module>
    locale.setlocale(locale.LC_ALL, 'ANSI_X3.4-1968')
  File "/home/travis/miniconda/envs/pandas/lib/python3.5/locale.py", line 594, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
```

What locale should we use?

@jreback I have searched! The thing is, all `xx_XX.encoding` have encodings that can tolerate some accents: 8859-1, UTF-8... Still, I'm going to try with 8859-1, just in case.

I doubt `zh_CN.UTF-8` would work, as I think it should set the encoding to utf-8, which is exactly what we need to avoid.

BTW, is it not wrong to assume that all locale settings match this pattern, `xx_XX.encoding` ? A user could set their locale to `C`. Maybe we can edit

```
  File "/home/travis/build/pydata/pandas/pandas/tools/tests/test_util.py", line 72, in test_set_locale
    lang, enc = LOCALE_OVERRIDE.split('.')
```

I was giving I am example of what other tests use

the locale utility is only for testing 
u can change it to work with C I think

```
$ ci/script.sh
inside ci/script.sh
ci/script.sh: line 13: warning: setlocale: LC_ALL: cannot change locale (en_US.iso88591): No such file or directory
```

Which locale are available, then?

no idea
maybe this: http://stackoverflow.com/questions/14547631/python-locale-error-unsupported-locale-setting

@jreback is this an acceptable solution?

can you update & rebase

@jreback all done!

As soon as the tests are confirmed, I'll correct the bug #12337

thanks!

ok if you'd submit on the fix for the issue would be great!