TST: Create compressed salary testing data
State: closed

Create compressed versions of the salary dataset for testing #14576.

Rename `salary.table.csv` to `salaries.tsv` because the dataset is tab rather than comma delimited. Remove the word table because it's implied by the extension. Rename `salary.table.gz` to `salaries.tsv.gz`, since compressed files should append to not strip the original extension.

Created new files by running the following commands:

```sh
cd pandas/io/tests/parser/data
bzip2 --keep salaries.tsv
xz --keep salaries.tsv
zip salaries.tsv.zip salaries.tsv
```

There will be CI testing failures since the modified files are not yet in master or on amazon S3. @jorisvandenbossche how do we deal with this?

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO CSV
description: read_csv, to_csv

Comments:
don't use tsv for extension,  use csv

> don't use tsv for extension, use csv

Why is that? The table isn't comma separated. This leads to issues like GitHub failing to render [the table](https://github.com/pandas-dev/pandas/blob/7a2bcb6605bacea858ec14cfac424898deb568b3/pandas/io/tests/parser/data/salary.table.csv):

![salaries-csv](https://cloud.githubusercontent.com/assets/1117703/20012377/529240ac-a285-11e6-9fcf-063ec7f8d290.png)

because you are changing something that is standard

if you wish to do that make a new PR but to be honest it's pretty non standard

> because you are changing something that is standard

@jreback hmm. Presently, the CSV standard is being violated as [RFC 4180](https://tools.ietf.org/html/rfc4180) specifies:

> Within the header and each record, there may be one or more fields, _separated by commas_.

However, `salary.table.csv` does meet the [TSV standard](https://www.iana.org/assignments/media-types/text/tab-separated-values):

> Fields in a record are separated from each other by a _tab character_.

So I'm happy to save this for another pull request, but I thought it would be most efficient to address it here... especially since:
1. we're already standardizing the file names in other ways -- like removing `.table` and adding the extension to proceed `.gz`.
2. changing the name of testing files creates temporary testing failures... so might as well do two birds with one stone?

Just let me know what's the best way forward.

Tagging @jreback @sinhrks @jorisvandenbossche -- let me know what to do next here, so I can continue with  #14576.

@dhimmel 
- let's keep the files as `.csv` for now. Although those files meet the tsv standard, this standard is certainly not followed in all fields, and also in the pandas codebase we use the `.csv` file extension for all text files at the moment, no matter the delimiter that is used.
- standardizing the file names in the other ways is certainly fine! (the compression suffixes and leaving out the 'table')

For the CI failures, those are indeed not avoidable. But since it doesn't change code, only the file names, it's OK if you can confirm those specific tests pass locally.

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14587?src=pr) is 85.27% (diff: 100%)

> Merging [#14587](https://codecov.io/gh/pandas-dev/pandas/pull/14587?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #14587   diff @@
==========================================
  Files           140        140          
  Lines         50693      50693          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43230      43229     -1   
- Misses         7463       7464     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [7a2bcb6...759d48d](https://codecov.io/gh/pandas-dev/pandas/compare/7a2bcb6605bacea858ec14cfac424898deb568b3...759d48d3b070764ab235f75e1acb5960f4d9a9ee?src=pr)

> let's keep the files as `.csv` for now.

Done in 24341b53341455433abcb6d01a2c7b4071e35316.

> For the CI failures, those are indeed not avoidable. But since it doesn't change code, only the file names, it's OK if you can confirm those specific tests pass locally.

@jorisvandenbossche, here're the errors I get when I run `nosetests pandas` locally in the development environment ([full log](https://github.com/pandas-dev/pandas/files/581222/nosetests.log.txt)):

```
ERROR: test_url_gz (pandas.io.tests.parser.test_network.TestUrlGz)
ERROR: test_url_gz_infer (pandas.io.tests.parser.test_network.TestUrlGz)
ERROR: test_url (pandas.io.tests.parser.test_parsers.TestCParserHighMemory)
ERROR: test_url (pandas.io.tests.parser.test_parsers.TestCParserLowMemory)
ERROR: test_url (pandas.io.tests.parser.test_parsers.TestPythonParser)
```

It appears to me that all errors are from remote URLs not resolving... but how do I "confirm these specific tests pass locally". Should I locally change the URLs?

> It appears to me that all errors are from remote URLs not resolving... but how do I "confirm these specific tests pass locally". Should I locally change the URLs?

Ah, yes, of course you have locally the same problem .. No problem, it's ok like this.

@dhimmel Thanks!

I made a small correction: https://github.com/pandas-dev/pandas/commit/ed2173695d6e5a2c69db3168d92d55327cd07276

> I made a small correction: ed21736

Yeah I should have more explicit that https://s3.amazonaws.com/pandas-test/salary.table.gz would have to be renamed on S3.

So in the longer term, here's what I'm thinking. First, is the [S3 test](https://github.com/pandas-dev/pandas/blob/ed2173695d6e5a2c69db3168d92d55327cd07276/pandas/io/tests/parser/test_network.py#L31-L35) actually valuable in addition to the GitHub URL test?
- If yes, then we should add the four compressed variants of `salaries.csv` to S3 using the new filename.
- If no, then we should only test the GitHub URLs, thereby removing the S3 (non-automated) dependency. As of 272669bec9fbe222ee5276f69806cb08ba6de1a6, #14576 removes the S3 test.

So if the S3 tests are valuable, then I'd suggest uploading `salaries.csv*` to S3 and I'll add those tests to #14576 accordingly.

@dhimmel that sounds right (the S3 here is just an url (not an S3 test), so github url is just as fine)

> @dhimmel that sounds right (the S3 here is just an url (not an S3 test), so github url is just as fine)

@jorisvandenbossche, based on how the reading from URL works, we actually will need to also test reading compressed files from S3 URLs. See https://github.com/pandas-dev/pandas/pull/14576/commits/bff0f3e2cdabfe02cfcbe95667a1a2ea1481fe1e

So would it be possible to upload `salaries.csv`, `salaries.csv.bz2`, `salaries.csv.gz`, `salaries.csv.xz`, and `salaries.csv.zip` to the S3?
@jreback Do you know who has access to the amazon data?

@dhimmel Are you that it is not just an url in this case? (the link you gave does not lead to visible code ("We went looking everywhere, but couldn’t find those commits"), probably because you had to force push after a rebase or so)



> probably because you had to force push after a rebase or so

Yep I rebased. See https://github.com/pandas-dev/pandas/pull/14576/commits/c0423918f1740f4caa03fe85db563d3837f7f4a8. The point is that a different implementation is used to read URLs that are S3.
Yes, but I think this url is not seen as a 'S3 url', but just as a plain url (if you enter this url in a browser, you just get to download the file). A S3 url would more look like `"s3://pandas-test/somefile.csv"`. What you link to is in the `s3.py` file, not in the main parsing  file.
For example:

```
In [14]: pd.io.common._is_s3_url('https://s3.amazonaws.com/pandas-test/salary.table.gz')
Out[14]: False

In [15]: pd.io.common._is_s3_url('s3://pandas-test/salary.table.gz')
Out[15]: True
```