Repository: sunpy/sunpy

Issue:
VSO QueryResponse should show all columns of results
Now we have `.show()` and such it makes no sense to limit the metadata we are showing the users, where we no longer do for all the header info in JSOC for example

Discussion:
pep8speaks: Hello @Cadair! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`sunpy/net/vso/vso.py`](https://github.com/sunpy/sunpy/blob/c601b0ce1289ec21f820e68adc26bba91f29ce29/sunpy/net/vso/vso.py):

> [Line 276:101](https://github.com/sunpy/sunpy/blob/c601b0ce1289ec21f820e68adc26bba91f29ce29/sunpy/net/vso/vso.py#L276): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (119 > 100 characters)


Cadair: If this passes all the tests, the tests are inadequate :rofl: 

Patch:
diff --git a/changelog/4788.bugfix.rst b/changelog/4788.bugfix.rst
new file mode 100644
index 000000000..869d0758f
--- /dev/null
+++ b/changelog/4788.bugfix.rst
@@ -0,0 +1,1 @@
+All columns from a :meth:`sunpy.net.vso.vso.VSOClient.search` will now be shown.
diff --git a/docs/guide/acquiring_data/fido.rst b/docs/guide/acquiring_data/fido.rst
index ee21c83d9..fb3ef9a5d 100644
--- a/docs/guide/acquiring_data/fido.rst
+++ b/docs/guide/acquiring_data/fido.rst
@@ -152,30 +152,30 @@ client for results, in this case VSO.::
     Results from 1 Provider:
     <BLANKLINE>
     289 Results from the VSOClient:
-        Start Time [1]       End Time [1]    Source ...   Type   Wavelength [2]
-                                                    ...             Angstrom
-     ------------------- ------------------- ------ ... -------- --------------
-     2012-03-04 00:00:00 2012-03-04 00:00:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-04 00:10:00 2012-03-04 00:10:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-04 00:20:00 2012-03-04 00:20:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-04 00:30:00 2012-03-04 00:30:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-04 00:40:00 2012-03-04 00:40:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-04 00:50:00 2012-03-04 00:50:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-04 01:00:00 2012-03-04 01:00:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-04 01:10:00 2012-03-04 01:10:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-04 01:20:00 2012-03-04 01:20:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-04 01:30:00 2012-03-04 01:30:01    SDO ... FULLDISK 171.0 .. 171.0
-                     ...                 ...    ... ...      ...            ...
-     2012-03-05 22:30:00 2012-03-05 22:30:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-05 22:40:00 2012-03-05 22:40:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-05 22:50:00 2012-03-05 22:50:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-05 23:00:00 2012-03-05 23:00:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-05 23:10:00 2012-03-05 23:10:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-05 23:20:00 2012-03-05 23:20:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-05 23:30:00 2012-03-05 23:30:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-05 23:40:00 2012-03-05 23:40:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-05 23:50:00 2012-03-05 23:50:01    SDO ... FULLDISK 171.0 .. 171.0
-     2012-03-06 00:00:00 2012-03-06 00:00:01    SDO ... FULLDISK 171.0 .. 171.0
+           Start Time       ...
+                            ...
+    ----------------------- ...
+    2012-03-04 00:00:00.000 ...
+    2012-03-04 00:10:00.000 ...
+    2012-03-04 00:20:00.000 ...
+    2012-03-04 00:30:00.000 ...
+    2012-03-04 00:40:00.000 ...
+    2012-03-04 00:50:00.000 ...
+    2012-03-04 01:00:00.000 ...
+    2012-03-04 01:10:00.000 ...
+    2012-03-04 01:20:00.000 ...
+    2012-03-04 01:30:00.000 ...
+                        ... ...
+    2012-03-05 22:30:00.000 ...
+    2012-03-05 22:40:00.000 ...
+    2012-03-05 22:50:00.000 ...
+    2012-03-05 23:00:00.000 ...
+    2012-03-05 23:10:00.000 ...
+    2012-03-05 23:20:00.000 ...
+    2012-03-05 23:30:00.000 ...
+    2012-03-05 23:40:00.000 ...
+    2012-03-05 23:50:00.000 ...
+    2012-03-06 00:00:00.000 ...
     Length = 289 rows
     <BLANKLINE>
     <BLANKLINE>
@@ -201,12 +201,12 @@ operator would::
     2012-03-04 00:00:00 2012-03-04 23:59:59     RHESSI ... RHESSI     NASA
     <BLANKLINE>
     3 Results from the VSOClient:
-       Start Time [1]       End Time [1]    Source ...     Type    Wavelength [2]
-                                                   ...                  keV
-    ------------------- ------------------- ------ ... ----------- --------------
-    2012-03-03 22:57:40 2012-03-04 00:33:20 RHESSI ... PARTIAL_SUN 3.0 .. 17000.0
-    2012-03-04 00:33:20 2012-03-04 01:45:40 RHESSI ... PARTIAL_SUN 3.0 .. 17000.0
-    2012-03-04 01:45:40 2012-03-04 02:09:00 RHESSI ... PARTIAL_SUN 3.0 .. 17000.0
+           Start Time               End Time        ... Extent Type      Info
+                                                    ...
+    ----------------------- ----------------------- ... ----------- --------------
+    2012-03-03 22:57:40.000 2012-03-04 00:33:20.000 ... PARTIAL_SUN RHESSI level-0
+    2012-03-04 00:33:20.000 2012-03-04 01:45:40.000 ... PARTIAL_SUN RHESSI level-0
+    2012-03-04 01:45:40.000 2012-03-04 02:09:00.000 ... PARTIAL_SUN RHESSI level-0
     <BLANKLINE>
     <BLANKLINE>
 
diff --git a/examples/acquiring_data/fido_metadata_queries.py b/examples/acquiring_data/fido_metadata_queries.py
index 445f64554..ccc974d34 100644
--- a/examples/acquiring_data/fido_metadata_queries.py
+++ b/examples/acquiring_data/fido_metadata_queries.py
@@ -5,7 +5,7 @@ Querying Metadata clients using Fido
 
 This example shows how to search and retrieve metadata using `~sunpy.net.Fido` from
 search facilities like `~sunpy.net.hek.HEKClient`, `~sunpy.net.helio.HECClient`,
-and `~sunpy.net.jsoc.JSOCClient`. It also shows how to display desired columns from the result.
+and `~sunpy.net.jsoc.JSOCClient`.
 """
 import os
 
@@ -13,11 +13,10 @@ from sunpy.net import Fido
 from sunpy.net import attrs as a
 
 ###################################################################
-# We will query the Heliophysics Integrated Observatory (`HELIO <https://www.helio-vo.eu/>`_) for the 'rhessi_flare_list' table.
+# We will query the `Heliophysics Integrated Observatory  <https://www.helio-vo.eu/>`__ for the 'rhessi_flare_list' table.
 # For the same time range, we will query HEK for 'FL' as the
 # event type and 'PeakFlux' greater than 1000.
 # We will also search JSOC for a 'hmi.m_45s' series.
-
 timerange = a.Time('2010/8/1 03:40', '2010/8/1 3:40:10')
 # Note that JSOC needs an email address, if you want to run this, you
 # must supply your own email.
diff --git a/sunpy/net/helio/hec.py b/sunpy/net/helio/hec.py
index e322d6b68..9be6ac22d 100644
--- a/sunpy/net/helio/hec.py
+++ b/sunpy/net/helio/hec.py
@@ -86,7 +86,7 @@ class HECClient(BaseClient):
             link = parser.wsdl_retriever()
         # Disable SSL check.
         session = Session()
-        session.verify = False
+        session.verify = True
         transport = Transport(session=session)
         self.hec_client = Client(link, transport=transport)
 
diff --git a/sunpy/net/vso/vso.py b/sunpy/net/vso/vso.py
index 6fcc903b5..03e1d98e8 100644
--- a/sunpy/net/vso/vso.py
+++ b/sunpy/net/vso/vso.py
@@ -67,6 +67,7 @@ class HashableResponse:
     ----------
     * https://stackoverflow.com/a/1305682
     """
+
     def __init__(self, d):
         self.data = d
         for a, b in d.items():
@@ -272,60 +273,58 @@ class QueryResponse(BaseQueryResponse):
                          max(record.time.end for record in self if record.time.end is not None))
 
     def build_table(self):
-        """
-        Create a human readable table.
-
-        Returns
-        -------
-        table : `astropy.table.QTable`
-        """
-        keywords = ['Start Time', 'End Time', 'Source', 'Instrument', 'Type', 'Wavelength']
-        record_items = {}
-        for key in keywords:
-            record_items[key] = []
-
-        def validate_time(time):
-            # Handle if the time is None when coming back from VSO
-            if time is None:
-                return ['None']
-            if record.time.start is not None:
-                return [parse_time(time).strftime(TIME_FORMAT)]
-            else:
-                return ['N/A']
-
-        for record in self:
-            record_items['Start Time'].append(validate_time(record.time.start))
-            record_items['End Time'].append(validate_time(record.time.end))
-            record_items['Source'].append(str(record.source))
-            record_items['Instrument'].append(str(record.instrument))
-            record_items['Type'].append(str(record.extent.type)
-                                        if record.extent.type is not None else ['N/A'])
-            # If we have a start and end Wavelength, make a quantity
-            if hasattr(record, 'wave') and record.wave.wavemin and record.wave.wavemax:
-                unit = record.wave.waveunit
-                # Convert this so astropy units parses it correctly
-                if unit == "kev":
-                    unit = "keV"
-                record_items['Wavelength'].append(u.Quantity([float(record.wave.wavemin),
-                                                              float(record.wave.wavemax)],
-                                                             unit=unit))
-            # If not save None
-            else:
-                record_items['Wavelength'].append(None)
-        # If we have no wavelengths for the whole list, drop the col
-        if all([a is None for a in record_items['Wavelength']]):
-            record_items.pop('Wavelength')
-            keywords.remove('Wavelength')
-        else:
-            # Make whole column a quantity
-            try:
-                with u.set_enabled_equivalencies(u.spectral()):
-                    record_items['Wavelength'] = u.Quantity(record_items['Wavelength'])
-            # If we have mixed units or some Nones just represent as strings
-            except (u.UnitConversionError, TypeError):
-                record_items['Wavelength'] = [str(a) for a in record_items['Wavelength']]
-
-        return Table(record_items)[keywords]
+        data = defaultdict(list)
+        # To maintain some compatibility with old versions, and to put the relevant info first,
+        # create the core columns
+        [data[k] for k in ['Start Time', 'End Time', 'Source', 'Instrument', 'Type', 'Wavelength']]
+        for record in self._data:
+            for key, value in record.data.items():
+                if not isinstance(value, dict):
+                    if key == "size":
+                        if value == -1:
+                            value = None
+                        else:
+                            # size is in bytes with a very high degree of precision.
+                            value = (value * u.Kibyte).to(u.Mibyte).round(5)
+                    data[key.title()].append(value)
+                else:
+                    if key == "wave":
+                        # Assumption here is that if min is empty, they all are.
+                        if value['wavemin'] is None:
+                            continue
+
+                        # Some records in the VSO have 'kev' which astropy
+                        # doesn't recognise as a unit, so fix it.
+                        waveunit = value['waveunit']
+                        waveunit = 'keV' if waveunit == 'kev' else waveunit
+
+                        data["Wavelength"].append(
+                            u.Quantity(
+                                [float(value['wavemin']), float(value['wavemax'])],
+                                unit=waveunit)
+                        )
+                        data["Wavetype"].append(value['wavetype'])
+                        continue
+                    for subkey, subvalue in value.items():
+                        key_template = f"{key.capitalize()} {subkey.capitalize()}"
+                        if key == "time" and subvalue is not None:
+                            key_template = f"{subkey.capitalize()} {key.capitalize()}"
+                            subvalue = parse_time(subvalue)
+                            # Change the display to the 'T'-less version
+                            subvalue.format = 'iso'
+                        data[key_template].append(subvalue)
+
+        # Remove Fileid as this method is only used for display.
+        if "Fileid" in data.keys():
+            data.pop("Fileid")
+
+        # Remove any columns which have no non-None values in them
+        clean_data = {}
+        for key, value in data.items():
+            if not all([x is None for x in value]):
+                clean_data[key] = value
+
+        return Table(clean_data)
 
     def add_error(self, exception):
         self.errors.append(exception)
@@ -385,7 +384,8 @@ class VSOClient(BaseClient):
         return obj(**kwargs)
 
     def search(self, *query):
-        """ Query data from the VSO with the new API. Takes a variable number
+        """
+        Query data from the VSO with the new API. Takes a variable number
         of attributes as parameter, which are chained together using AND.
 
         The new query language allows complex queries to be easily formed.
@@ -402,20 +402,20 @@ class VSOClient(BaseClient):
         ...    a.Time(datetime(2010, 1, 1), datetime(2010, 1, 1, 1)),
         ...    a.Instrument.eit | a.Instrument.aia)   # doctest:  +REMOTE_DATA
         <sunpy.net.vso.vso.QueryResponse object at ...>
-           Start Time [1]       End Time [1]    Source ...   Type   Wavelength [2]
-                                                       ...             Angstrom
-        ------------------- ------------------- ------ ... -------- --------------
-        2010-01-01 00:00:08 2010-01-01 00:00:20   SOHO ... FULLDISK 195.0 .. 195.0
-        2010-01-01 00:12:08 2010-01-01 00:12:20   SOHO ... FULLDISK 195.0 .. 195.0
-        2010-01-01 00:24:10 2010-01-01 00:24:22   SOHO ... FULLDISK 195.0 .. 195.0
-        2010-01-01 00:36:08 2010-01-01 00:36:20   SOHO ... FULLDISK 195.0 .. 195.0
-        2010-01-01 00:48:09 2010-01-01 00:48:21   SOHO ... FULLDISK 195.0 .. 195.0
+               Start Time               End Time        Source ... Extent Type   Size
+                                                               ...              Mibyte
+        ----------------------- ----------------------- ------ ... ----------- -------
+        2010-01-01 00:00:08.000 2010-01-01 00:00:20.000   SOHO ...    FULLDISK 2.01074
+        2010-01-01 00:12:08.000 2010-01-01 00:12:20.000   SOHO ...    FULLDISK 2.01074
+        2010-01-01 00:24:10.000 2010-01-01 00:24:22.000   SOHO ...    FULLDISK 2.01074
+        2010-01-01 00:36:08.000 2010-01-01 00:36:20.000   SOHO ...    FULLDISK 2.01074
+        2010-01-01 00:48:09.000 2010-01-01 00:48:21.000   SOHO ...    FULLDISK 2.01074
 
         Returns
         -------
-        out : :py:class:`QueryResult` (enhanced list)
+        out : `QueryResult`
             Matched items. Return value is of same type as the one of
-            :py:meth:`VSOClient.search`.
+            :meth:`VSOClient.search`.
         """
         query = and_(*query)
         QueryRequest = self.api.get_type('VSO:QueryRequest')


Test patch:
diff --git a/sunpy/net/vso/tests/test_vso.py b/sunpy/net/vso/tests/test_vso.py
index ef61082cc..ab1983c9e 100644
--- a/sunpy/net/vso/tests/test_vso.py
+++ b/sunpy/net/vso/tests/test_vso.py
@@ -24,11 +24,10 @@ class MockQRRecord:
 
     def __init__(self, start_time=None, end_time=None, size=0, source='SOHO', instrument='aia',
                  extent_type=None):
-        self.size = size
-        self.time = MockObject(start=start_time, end=end_time)
-        self.source = a.Source(source)
-        self.instrument = core_attrs.Instrument(instrument)
-        self.extent = MockObject(type=None if extent_type is None else extent_type.type)
+        self.data = {"size": size, "time": dict(start=start_time, end=end_time),
+                     "source": a.Source(source),
+                     "instrument": core_attrs.Instrument(instrument),
+                     "extent": dict(type=None if extent_type is None else extent_type.type)}
 
 
 class MockQRResponse:
@@ -258,22 +257,20 @@ def test_wave_repr():
 @mock.patch("sunpy.net.vso.vso.build_client", return_value=True)
 def test_str(mock_build_client):
     qr = QueryResponse([])
-    assert str(qr) == ('Start Time End Time Source Instrument Type\n'
-                       '---------- -------- ------ ---------- ----')
+    assert str(qr) == '<No columns>'
 
 
 @mock.patch("sunpy.net.vso.vso.build_client", return_value=True)
 def test_repr(mock_build_client):
     qr = QueryResponse([])
-    assert "Start Time End Time Source Instrument Type" in repr(qr)
+    assert '<No columns>' in repr(qr)
 
 
 @mock.patch("sunpy.net.vso.vso.build_client", return_value=True)
 def test_show(mock_build_client):
     qr = QueryResponse([])
     qrshow = qr.show('Start Time', 'Source', 'Type')
-    assert str(qrshow) == ('Start Time Source Type\n'
-                           '---------- ------ ----')
+    assert str(qrshow) == '<No columns>'
 
 
 @pytest.mark.remote_data
@@ -368,40 +365,30 @@ def test_QueryResponse_build_table_defaults(mock_build_client):
     qr = vso.QueryResponse(records)
     table = qr.build_table()
 
-    start_time_ = table['Start Time']
-    assert len(start_time_) == 1
-    assert start_time_[0] == 'None'
-
-    end_time_ = table['End Time']
-    assert len(end_time_) == 1
-    assert end_time_[0] == 'None'
-
-    type_ = table['Type'].data
-    assert len(type_) == 1
-    assert type_[0] == 'N/A'
-
-    # Check values we did set by default in 'MockQRRecord'
+    # These are the only None values in the table.
     source_ = table['Source'].data
     assert len(source_) == 1
-    assert source_[0][:-16] == str(a.Source('SOHO'))[:-16]
+    assert source_[0].value == 'SOHO'
 
     instrument_ = table['Instrument'].data
     assert len(instrument_) == 1
-    assert instrument_[0][:-16] == str(core_attrs.Instrument('aia'))[:-16]
+    assert instrument_[0].value == 'aia'
+
+    size_ = table['Size'].data
+    assert len(size_) == 1
+    assert size_[0] == 0.0
 
 
 @mock.patch("sunpy.net.vso.vso.build_client", return_value=True)
 def test_QueryResponse_build_table_with_extent_type(mock_build_client):
     """
-    When explcitley suppling an 'Extent' only the 'type' is stored
+    When explicitly suppling an 'Extent' only the 'type' is stored
     in the built table.
     """
     e_type = va.Extent(x=1.0, y=2.5, width=37, length=129.2, atype='CORONA')
-
     qr = vso.QueryResponse((MockQRRecord(extent_type=e_type),))
     table = qr.build_table()
-
-    extent = table['Type'].data
+    extent = table['Extent Type'].data
     assert len(extent) == 1
     assert extent[0] == e_type.type
 
@@ -418,15 +405,11 @@ def test_QueryResponse_build_table_with_no_start_time(mock_build_client):
     qr = vso.QueryResponse(records)
     table = qr.build_table()
 
-    start_time_ = table['Start Time']
-    assert len(start_time_) == 1
-    assert start_time_[0] == 'None'
-
-    # Even though 'End Time' is valid, there is no 'Start Time'
-    # marks as 'N/A'
+    # 'End Time' is valid, there is no 'Start Time' in the table
+    assert 'Start Time' not in table.columns
     end_time_ = table['End Time']
     assert len(end_time_) == 1
-    assert end_time_[0] == 'N/A'
+    assert end_time_[0].value == '2016-02-14 08:08:12.000'
 
 
 @mock.patch("sunpy.net.vso.vso.build_client", return_value=True)
@@ -440,14 +423,9 @@ def test_QueryResponse_build_table_with_no_end_time(mock_build_client):
 
     qr = vso.QueryResponse(records)
     table = qr.build_table()
-
     start_time_ = table['Start Time']
     assert len(start_time_) == 1
-    assert start_time_[0] == '2016-02-14 08:08:12'
-
-    end_time_ = table['End Time']
-    assert len(end_time_) == 1
-    assert end_time_[0] == 'None'
+    assert start_time_[0].value == '2016-02-14 08:08:12.000'
 
 
 @pytest.mark.remote_data