Repository: astropy/astropy

Issue:
Feature request: command-line tool to view binary tables
As discussed in this Twitter conversation: https://twitter.com/alexavillaume/status/518096636313346049 - the world would be a better place if we all used binary tables instead of ASCII, but the issue is that ASCII tables are so damn convenient to view quickly. But what if we created a simple command-line script that basically uses `Table.read` to read in a binary table, then uses `Table.more()` to display it?

```
$ table-more mytable.xml
           col1            col2    col3    col4  ... col7  col8 col9 col10
------------------------- ------ ------- ------- ... ---- ----- ---- -----
SSTGLMC G000.0046+01.1431 0.0046  1.1432 265.299 ... 5.04  6.89 5.22     N
SSTGLMC G000.0106-00.7315 0.0106 -0.7314 267.127 ... 6.07    -- 5.17     Y
SSTGLMC G000.0110-01.0237  0.011 -1.0236 267.415 ...  6.3  8.34 6.32     N
SSTGLMC G000.0115-01.2781 0.0115  -1.278 267.666 ... 6.84   8.3 6.87     N
SSTGLMC G000.0127-01.2146 0.0127 -1.2146 267.604 ... 5.53  6.58 5.51     N
SSTGLMC G000.0177-01.8424 0.0177 -1.8423 268.229 ... 8.04  10.1 8.04     N
SSTGLMC G000.0371+01.6473 0.0371  1.6475 264.834 ...  5.2    --  4.7     Y
SSTGLMC G000.0408-00.7197 0.0408 -0.7196 267.134 ...  6.0 10.77 5.99     N
SSTGLMC G000.0540-00.7328  0.054 -0.7327 267.154 ... 6.91  8.09  6.9     N
SSTGLMC G000.0566-00.7363 0.0567 -0.7363 267.159 ...  7.9   9.4 7.83     N
SSTGLMC G000.0570+01.2494 0.0571  1.2495 265.228 ... 8.88 10.37 8.95     N
```

It could of course take options to specify the table ID/name in multi-table files. Could be pretty handy!


Discussion:
astropy-bot[bot]: Hi there @saimn :wave: - thanks for the pull request! I'm just a friendly :robot: that checks for issues related to the changelog and making sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part :smiley:.

I noticed the following issues with this pull request:

* The milestone has not been set (this can only be set by a maintainer)
* Changelog entry not present (or pull request number missing) and neither the **Affects-dev** nor the **no-changelog-entry-needed** label are set

Would it be possible to fix these? Thanks!

*If there are any issues with this message, please report them [here](https://github.com/astropy/astropy-bot/issues).*
saimn: Missing: 
- [ ] changelog entry
- [ ] mention in docs

Note that I created the file inside a new directory `astropy/table/scripts/showtable.py` as it looks logical but I can move it if needed.
pllim: c/c @SaraOgaz in case this is of interest for STAK.
taldcroft: Also, you could probably use this interface to print the results of `table.info` (with an `--info` flag?).

saimn: Updated `--help` output with the new parameters (`--delimiter`, `--info` and `--stats`), I think it's still reasonably short:
```
❯ showtable --help
usage: showtable [-h] [--more] [--info] [--stats] [--max-lines MAX_LINES]
                 [--max-width MAX_WIDTH] [--hide-unit] [--show-dtype]
                 [--format FORMAT] [--delimiter DELIMITER] [--hdu HDU]
                 [--path PATH] [--table_id TABLE_ID]
                 filename [filename ...]

Print tables from ASCII, FITS, HDF5, VOTable file(s).The default behavior is
make the table output fit onto a single screen page. For a long and wide table
this will mean cutting out inner rows and columns. To print **all** the rows
or columns use ``--max-lines=-1`` or ``max-width=-1``, respectively.

positional arguments:
  filename              Path to one or more files.

optional arguments:
  -h, --help            show this help message and exit
  --more                Use the pager mode from Table.more.
  --info                Show information about the table columns.
  --stats               Show statistics about the table columns.
  --max-lines MAX_LINES
                        Maximum number of lines in table output
                        (default=screen length, -1 for no limit).
  --max-width MAX_WIDTH
                        Maximum width in table output (default=screen width,
                        -1 for no limit).
  --hide-unit           Hide the header row for unit (which is shown only if
                        one or more columns has a unit).
  --show-dtype          Include a header row for column dtypes.
  --format FORMAT       Input table format (only for ASCII files).
  --delimiter DELIMITER
                        Column delimiter string (only for ASCII files).
  --hdu HDU             Name of the HDU to show (only for FITS files).
  --path PATH           The path from which to read the table (only for HDF5
                        files).
  --table_id TABLE_ID   The table to read in (only for VOTable files).
```
saimn: I have also added a mention of the script in the unified I/O page  : 

![screenshot-2017-11-17 unified file read write interface astropy v3 0 dev20629](https://user-images.githubusercontent.com/311639/32972274-ce0f40ea-cbf1-11e7-8a0f-07acadf32fa1.png)
  
taldcroft: @saimn - About the subparser, or else generalized support for sub-format options, I think you need to think about this as a high-visibility script that will be around for 15 or 20 years.  

Pretty quickly we will add support for reading ASDF files, so that will result in at least one (if not more) new params.  And with the mem-mapping in FITS it would make sense to add support in the API for reading in only some of the columns.  Then a user will want the `astropy_native` option, and another will say that they have an ASCII file with `%` for comments, and another will say that they have a file that starts on the 5th line, and so on.

So I think that it is worth thinking hard about this script and future-proofing the UI, even if it means typing `showtable fits filename.fits` instead of `showtable filename.fits`.  Or maybe there is another option (subformat required only to specify subformat options?), but I think that having a flat set of options that handles *every* file reader for the next 15-20 years will get messy.
saimn: @taldcroft , your arguments make perfectly sense, (and I agree it's good to think about this now before the is widely used), despite that I would prefer to avoid subparsers. I think subparsers makes the command more complicated than what it should be, you can read a table with `Table.read('foo.fits')` or `Table.read('foo.csv')`, it should be as simple for the script. 
But that's just my own preference, I can live with subparsers if I really have to :wink:, what do others think ?
taldcroft: @saimn - agreed that we need to find a balance between overdesigning and getting into a mess down the road.  I too am interested in other opinions.

If you put in a general mechanism for handling any kwargs that are not parsed by argparse, then you get back to the simplicity but full generality of `Table.read()`.  What you lose is documentation of available kwargs, but this is a long-standing problem with `Table.read()` as well so it would be OK.
saimn: I have used [argument groups](https://docs.python.org/dev/library/argparse.html#argument-groups) to group arguments for each filetype, which gives a much better output I think (see below) and would allow to add a few more arguments while not being too messy. 

What do you think, compared to the sub-parsers option mentioned above, @taldcroft (and others - @astrofrog , @pllim , ... ? :wink: )

```
❯ showtable --help
usage: showtable [-h] [--more] [--info] [--stats] [--max-lines MAX_LINES]
                 [--max-width MAX_WIDTH] [--hide-unit] [--show-dtype]
                 [--format FORMAT] [--delimiter DELIMITER] [--hdu HDU]
                 [--path PATH] [--table-id TABLE_ID]
                 filename [filename ...]

Print tables from ASCII, FITS, HDF5, VOTable file(s). The tables are read with
'astropy.table.Table.read' and are printed with 'astropy.table.Table.pprint'.
The default behavior is to make the table output fit onto a single screen
page. For a long and wide table this will mean cutting out inner rows and
columns. To print **all** the rows or columns use ``--max-lines=-1`` or ``max-
width=-1``, respectively.

positional arguments:
  filename              path to one or more files

optional arguments:
  -h, --help            show this help message and exit
  --more                use the pager mode from Table.more
  --info                show information about the table columns
  --stats               show statistics about the table columns

pprint arguments:
  --max-lines MAX_LINES
                        maximum number of lines in table output
                        (default=screen length, -1 for no limit)
  --max-width MAX_WIDTH
                        maximum width in table output (default=screen width,
                        -1 for no limit)
  --hide-unit           hide the header row for unit (which is shown only if
                        one or more columns has a unit)
  --show-dtype          include a header row for column dtypes

ASCII arguments:
  --format FORMAT       input table format
  --delimiter DELIMITER
                        column delimiter string

FITS arguments:
  --hdu HDU             name of the HDU to show

HDF5 arguments:
  --path PATH           the path from which to read the table

VOTable arguments:
  --table-id TABLE_ID   the table to read in
```
taldcroft: Ah, I didn't know about argument groups.  That works for me and looks nice.

I wonder if `--format` should be global in case some file fails to be auto-identified.  E.g. a FITS file that doesn't have exactly the initial signature and an extension that doesn't match.
saimn: @taldcroft - Is it possible to force the Reader class used by the registry with a 'format' keyword ?

Also there is a test error with end of lines on appveyor, with Table.info's output. Why is it different from Table.pprint ?
pllim: @saimn , I like the grouping -- very nice! :+1: 
taldcroft: > Is it possible to force the Reader class used by the registry with a 'format' keyword ?

Are you talking about the Reader class in `io.ascii` or in the general unified I/O scheme?  For `io.ascii` the answer is yes, via the `format` keyword which takes the "nice" ASCII representation of the Reader class name (with the caveat that it is prepended by `'ascii.'` in unified I/O).  In the latter the reader class is determined from the caller, i.e. `Table.read` and this cannot be overridden (AFAIK).
saimn: I mean for the general case, if I have an exotic fits extension is it possible to use `format='fits'` ? I also think it is not possible currently, but wasn't sure given your sentence above - (*I wonder if --format should be global in case some file fails to be auto-identified. E.g. a FITS file*)
saimn: Also the appveyor issue is fixed, so all tests pass, and argument groups seems to have the consensus. Anything else that should be added/modified ?
saimn: @taldcroft - Thanks for your useful comments, I addressed all of it.
taldcroft: @saimn - looks good, except the two failing tests due to `bytes20` instead of `str20`.  This makes sense with recent changes to the FITS Table interface.  What doesn't make sense is why the circleci tests are *passing*.
saimn: @taldcroft - Hmm weird, that doesn't make sense to me since I did not rebase on the `Table.read` PR, and tests are passing locally (with `str20`)...
But I should rebase anyway, as it should indeed change the string columns dtype.

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 7c50a53a02..79dc32f618 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -113,6 +113,9 @@ astropy.table
 
 - Allowed to remove table rows through the ``__delitem__`` method. [#5839]
 
+- Added a new ``showtable`` command-line script to view binary or ASCII table
+  files. [#6859]
+
 astropy.tests
 ^^^^^^^^^^^^^
 
diff --git a/astropy/table/scripts/__init__.py b/astropy/table/scripts/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/astropy/table/scripts/showtable.py b/astropy/table/scripts/showtable.py
new file mode 100644
index 0000000000..3ff4005f24
--- /dev/null
+++ b/astropy/table/scripts/showtable.py
@@ -0,0 +1,158 @@
+# Licensed under a 3-clause BSD style license - see LICENSE.rst
+"""
+``showtable`` is a command-line script based on ``astropy.io`` and
+``astropy.table`` for printing ASCII, FITS, HDF5 or VOTable files(s) to the
+standard output.
+
+Example usage of ``showtable``:
+
+1. FITS::
+
+    $ showtable astropy/io/fits/tests/data/table.fits
+
+     target V_mag
+    ------- -----
+    NGC1001  11.1
+    NGC1002  12.3
+    NGC1003  15.2
+
+2. ASCII::
+
+    $ showtable astropy/io/ascii/tests/t/simple_csv.csv
+
+     a   b   c
+    --- --- ---
+      1   2   3
+      4   5   6
+
+3. XML::
+
+    $ showtable astropy/io/votable/tests/data/names.xml --max-width 70
+
+               col1             col2     col3  ... col15 col16 col17
+               ---              deg      deg   ...  mag   mag   ---
+    ------------------------- -------- ------- ... ----- ----- -----
+    SSTGLMC G000.0000+00.1611   0.0000  0.1611 ...    --    --    AA
+
+
+
+4. Print all the FITS tables in the current directory::
+
+    $ showtable *.fits
+
+"""
+
+import argparse
+import textwrap
+import warnings
+from astropy import log
+from astropy.table import Table
+from astropy.utils.exceptions import AstropyUserWarning
+
+
+def showtable(filename, args):
+    """
+    Read a table and print to the standard output.
+
+    Parameters
+    ----------
+    filename : str
+        The path to a FITS file.
+
+    """
+    if args.info and args.stats:
+        warnings.warn('--info and --stats cannot be used together',
+                      AstropyUserWarning)
+    if (any((args.max_lines, args.max_width, args.hide_unit, args.show_dtype))
+            and (args.info or args.stats)):
+        warnings.warn('print parameters are ignored if --info or --stats is '
+                      'used', AstropyUserWarning)
+
+    # these parameters are passed to Table.read if they are specified in the
+    # command-line
+    read_kwargs = ('hdu', 'format', 'table_id', 'delimiter')
+    kwargs = {k: v for k, v in vars(args).items()
+              if k in read_kwargs and v is not None}
+    try:
+        table = Table.read(filename, **kwargs)
+        if args.info:
+            table.info('attributes')
+        elif args.stats:
+            table.info('stats')
+        else:
+            formatter = table.more if args.more else table.pprint
+            formatter(max_lines=args.max_lines, max_width=args.max_width,
+                      show_unit=(False if args.hide_unit else None),
+                      show_dtype=args.show_dtype)
+    except IOError as e:
+        log.error(str(e))
+
+
+def main(args=None):
+    """The main function called by the `showtable` script."""
+    parser = argparse.ArgumentParser(
+        description=textwrap.dedent("""
+            Print tables from ASCII, FITS, HDF5, VOTable file(s).  The tables
+            are read with 'astropy.table.Table.read' and are printed with
+            'astropy.table.Table.pprint'. The default behavior is to make the
+            table output fit onto a single screen page.  For a long and wide
+            table this will mean cutting out inner rows and columns.  To print
+            **all** the rows or columns use ``--max-lines=-1`` or
+            ``max-width=-1``, respectively. The complete list of supported
+            formats can be found at
+            http://astropy.readthedocs.io/en/latest/io/unified.html#built-in-table-readers-writers
+        """))
+
+    addarg = parser.add_argument
+    addarg('filename', nargs='+', help='path to one or more files')
+
+    addarg('--format', help='input table format, should be specified if it '
+           'cannot be automatically detected')
+    addarg('--more', action='store_true',
+           help='use the pager mode from Table.more')
+    addarg('--info', action='store_true',
+           help='show information about the table columns')
+    addarg('--stats', action='store_true',
+           help='show statistics about the table columns')
+
+    # pprint arguments
+    pprint_args = parser.add_argument_group('pprint arguments')
+    addarg = pprint_args.add_argument
+    addarg('--max-lines', type=int,
+           help='maximum number of lines in table output (default=screen '
+           'length, -1 for no limit)')
+    addarg('--max-width', type=int,
+           help='maximum width in table output (default=screen width, '
+           '-1 for no limit)')
+    addarg('--hide-unit', action='store_true',
+           help='hide the header row for unit (which is shown '
+           'only if one or more columns has a unit)')
+    addarg('--show-dtype', action='store_true',
+           help='include a header row for column dtypes')
+
+    # ASCII-specific arguments
+    ascii_args = parser.add_argument_group('ASCII arguments')
+    addarg = ascii_args.add_argument
+    addarg('--delimiter', help='column delimiter string')
+
+    # FITS-specific arguments
+    fits_args = parser.add_argument_group('FITS arguments')
+    addarg = fits_args.add_argument
+    addarg('--hdu', help='name of the HDU to show')
+
+    # HDF5-specific arguments
+    hdf5_args = parser.add_argument_group('HDF5 arguments')
+    addarg = hdf5_args.add_argument
+    addarg('--path', help='the path from which to read the table')
+
+    # VOTable-specific arguments
+    votable_args = parser.add_argument_group('VOTable arguments')
+    addarg = votable_args.add_argument
+    addarg('--table-id', help='the table to read in')
+
+    args = parser.parse_args(args)
+
+    for idx, filename in enumerate(args.filename):
+        if idx > 0:
+            print()
+        showtable(filename, args)
diff --git a/docs/io/unified.rst b/docs/io/unified.rst
index 8e2f78fb4d..9e27e13ba2 100644
--- a/docs/io/unified.rst
+++ b/docs/io/unified.rst
@@ -58,6 +58,24 @@ Any additional arguments specified will depend on the format.  For examples of t
 section `Built-in table readers/writers`_.  This section also provides the full list of
 choices for the ``format`` argument.
 
+Command-line utility
+--------------------
+
+For convenience, the command-line tool ``showtable`` can be used to print the
+content of tables for the formats supported by the unified I/O interface::
+
+    $ showtable astropy/io/fits/tests/data/table.fits
+
+     target V_mag
+    ------- -----
+    NGC1001  11.1
+    NGC1002  12.3
+    NGC1003  15.2
+
+To get full documentation on the usage and available options do ``showtable
+--help``.
+
+
 .. _built_in_readers_writers:
 
 Built-in table readers/writers
diff --git a/setup.py b/setup.py
index 72ccd51fc6..074c050853 100755
--- a/setup.py
+++ b/setup.py
@@ -54,6 +54,7 @@ entry_points['console_scripts'] = [
     'fitsheader = astropy.io.fits.scripts.fitsheader:main',
     'fitsinfo = astropy.io.fits.scripts.fitsinfo:main',
     'samp_hub = astropy.samp.hub_script:hub_script',
+    'showtable = astropy.table.scripts.showtable:main',
     'volint = astropy.io.votable.volint:main',
     'wcslint = astropy.wcs.wcslint:main',
 ]


Test patch:
diff --git a/astropy/table/tests/test_showtable.py b/astropy/table/tests/test_showtable.py
new file mode 100644
index 0000000000..35b346a678
--- /dev/null
+++ b/astropy/table/tests/test_showtable.py
@@ -0,0 +1,156 @@
+import os
+
+from ..scripts import showtable
+
+ROOT = os.path.abspath(os.path.dirname(__file__))
+ASCII_ROOT = os.path.join(ROOT, '..', '..', 'io', 'ascii', 'tests')
+FITS_ROOT = os.path.join(ROOT, '..', '..', 'io', 'fits', 'tests')
+VOTABLE_ROOT = os.path.join(ROOT, '..', '..', 'io', 'votable', 'tests')
+
+
+def test_missing_file(capsys):
+    showtable.main(['foobar.fits'])
+    out, err = capsys.readouterr()
+    assert err.startswith("ERROR: [Errno 2] No such file or directory: "
+                          "'foobar.fits'")
+
+
+def test_info(capsys):
+    showtable.main([os.path.join(FITS_ROOT, 'data/table.fits'), '--info'])
+    out, err = capsys.readouterr()
+    assert out.splitlines() == ['<Table length=3>',
+                                ' name   dtype ',
+                                '------ -------',
+                                'target bytes20',
+                                ' V_mag float32']
+
+
+def test_stats(capsys):
+    showtable.main([os.path.join(FITS_ROOT, 'data/table.fits'), '--stats'])
+    out, err = capsys.readouterr()
+    assert out == ('<Table length=3>{0}'
+                   ' name    mean    std   min  max {0}'
+                   '------ ------- ------- ---- ----{0}'
+                   'target      --      --   --   --{0}'
+                   ' V_mag 12.8667 1.72111 11.1 15.2{0}').format(os.linesep)
+
+
+def test_fits(capsys):
+    showtable.main([os.path.join(FITS_ROOT, 'data/table.fits')])
+    out, err = capsys.readouterr()
+    assert out.splitlines() == [' target V_mag',
+                                '------- -----',
+                                'NGC1001  11.1',
+                                'NGC1002  12.3',
+                                'NGC1003  15.2']
+
+
+def test_fits_hdu(capsys):
+    showtable.main([os.path.join(FITS_ROOT, 'data/zerowidth.fits'),
+                    '--hdu', 'AIPS OF'])
+    out, err = capsys.readouterr()
+    assert out.startswith(
+        '  TIME   SOURCE ID ANTENNA NO. SUBARRAY FREQ ID ANT FLAG STATUS 1\n'
+        '  DAYS                                                           \n'
+        '-------- --------- ----------- -------- ------- -------- --------\n'
+        '0.144387         1          10        1       1        4        4\n'
+    )
+
+
+def test_csv(capsys):
+    showtable.main([os.path.join(ASCII_ROOT, 't/simple_csv.csv')])
+    out, err = capsys.readouterr()
+    assert out.splitlines() == [' a   b   c ',
+                                '--- --- ---',
+                                '  1   2   3',
+                                '  4   5   6']
+
+
+def test_ascii_format(capsys):
+    showtable.main([os.path.join(ASCII_ROOT, 't/commented_header.dat'),
+                    '--format', 'ascii.commented_header'])
+    out, err = capsys.readouterr()
+    assert out.splitlines() == [' a   b   c ',
+                                '--- --- ---',
+                                '  1   2   3',
+                                '  4   5   6']
+
+
+def test_ascii_delimiter(capsys):
+    showtable.main([os.path.join(ASCII_ROOT, 't/simple2.txt'),
+                    '--format', 'ascii', '--delimiter', '|'])
+    out, err = capsys.readouterr()
+    assert out.splitlines() == [
+        "obsid redshift  X    Y      object   rad ",
+        "----- -------- ---- ---- ----------- ----",
+        " 3102     0.32 4167 4085 Q1250+568-A  9.0",
+        " 3102     0.32 4706 3916 Q1250+568-B 14.0",
+        "  877     0.22 4378 3892 'Source 82' 12.5",
+    ]
+
+
+def test_votable(capsys):
+    showtable.main([os.path.join(VOTABLE_ROOT, 'data/regression.xml'),
+                    '--table-id', 'main_table', '--max-width', '50'])
+    out, err = capsys.readouterr()
+    assert out.splitlines() == [
+        '   string_test    string_test_2 ... bitarray2 [16]',
+        '----------------- ------------- ... --------------',
+        '    String & test    Fixed stri ...  True .. False',
+        'String &amp; test    0123456789 ...       -- .. --',
+        '             XXXX          XXXX ...       -- .. --',
+        '                                ...       -- .. --',
+        '                                ...       -- .. --',
+    ]
+
+
+def test_max_lines(capsys):
+    showtable.main([os.path.join(ASCII_ROOT, 't/cds2.dat'),
+                    '--format', 'ascii.cds', '--max-lines', '7',
+                    '--max-width', '30'])
+    out, err = capsys.readouterr()
+    assert out.splitlines() == [
+        '      SST       ... Note',
+        '                ...     ',
+        '--------------- ... ----',
+        '041314.1+281910 ...   --',
+        '            ... ...  ...',
+        '044427.1+251216 ...   --',
+        '044642.6+245903 ...   --',
+        'Length = 215 rows',
+    ]
+
+
+def test_show_dtype(capsys):
+    showtable.main([os.path.join(FITS_ROOT, 'data/table.fits'),
+                    '--show-dtype'])
+    out, err = capsys.readouterr()
+    assert out.splitlines() == [
+        ' target  V_mag ',
+        'bytes20 float32',
+        '------- -------',
+        'NGC1001    11.1',
+        'NGC1002    12.3',
+        'NGC1003    15.2',
+    ]
+
+
+def test_hide_unit(capsys):
+    showtable.main([os.path.join(ASCII_ROOT, 't/cds.dat'),
+                    '--format', 'ascii.cds'])
+    out, err = capsys.readouterr()
+    assert out.splitlines() == [
+        'Index RAh RAm  RAs  DE- DEd  DEm    DEs   Match Class  AK Fit ',
+        '       h  min   s       deg arcmin arcsec             mag     ',
+        '----- --- --- ----- --- --- ------ ------ ----- ----- --- ----',
+        '    1   3  28 39.09   +  31      6    1.9    --    I*  -- 1.35',
+    ]
+
+    showtable.main([os.path.join(ASCII_ROOT, 't/cds.dat'),
+                    '--format', 'ascii.cds', '--hide-unit'])
+    out, err = capsys.readouterr()
+    assert out.splitlines() == [
+        'Index RAh RAm  RAs  DE- DEd DEm DEs Match Class  AK Fit ',
+        '----- --- --- ----- --- --- --- --- ----- ----- --- ----',
+        '    1   3  28 39.09   +  31   6 1.9    --    I*  -- 1.35',
+    ]