Package: imap-data-access
IMAP Data Access PackageThis lightweight  Python package allows users to download, query, and upload data from the IMAP Science Data Center (SDC).Command Line UtilityQuery / Search for dataFind all files from the SWE instrument$imap-data-accessquery--instrumentswe
Found[2]matchingfiles
---------------------------------------------------------------------------------------------------------------|Instrument|DataLevel|Descriptor|StartDate|EndDate|Version|Filename|---------------------------------------------------------------------------------------------------------------|swe|l0|sci|20240105|20240105|v00-05|imap_swe_l0_sci_20240105_20240105_v00-05.pkts|swe|l0|sci|20240105|20240105|v00-01|imap_swe_l0_sci_20240105_20240105_v00-01.pkts|---------------------------------------------------------------------------------------------------------------|Find all files during the year 2024 and return the response as raw json$imap-data-accessquery--start-date20240101--end-date20241231--output-formatjson[{'file_path':'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-05.pkts','instrument':'swe','data_level':'l0','descriptor':'sci','start_date':'20240105','end_date':'20240105','version':'v00-05','extension':'pkts'},{'file_path':'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts','instrument':'swe','data_level':'l0','descriptor':'sci','start_date':'20240105','end_date':'20240105','version':'v00-01','extension':'pkts'}]Download a fileDownload a level 0 SWE file on 2024/01/05$imap-data-accessdownloadimap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts
Successfullydownloadedthefileto:<IMAP_DATA_DIR>/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pktsUpload a fileUpload a l1a file after decoding the l0 CCSDS ".pkts" file$imap-data-accessupload/imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_20240105_v00-00.cdf
SuccessfullyuploadedthefiletotheIMAPSDCImporting as a packageimportimap_data_access# Search for filesresults=imap_data_access.query(instrument="mag",data_level="l0")# results is a list of dictionaries# [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-05.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'end_date': '20240105', 'version': 'v00-05', 'extension': 'pkts'}, {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts', 'instrument': 'swe', 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'end_date': '20240105', 'version': 'v00-01', 'extension': 'pkts'}]# Download a file that was returned from the searchimap_data_access.download("imap/mag/l0/2024/01/imap_mag_l0_raw_202040101_20240101_v00-00.pkts")# Upload a calibration file that exists locallyimap_data_access.upload("imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_20240105_v00-00.cdf")ConfigurationData directoryThe folder structure for data files within the IMAP SDC is rigidly
defined, so the data access will mimic that structure to make sure
all data is stored in the same heirarchical structure as the SDC.
This will enable seamless transition between a user's local system
and the SDC.A user's root data location can be specified as an environment
variableIMAP_DATA_DIRor through a configuration dictionary
within the package itselfimap_data_access.config["DATA_DIR"].
If theIMAP_DATA_DIRvariable is not set, the program defaults
to the user's current working directory +data/.The following is the directory structure the IMAP SDC uses.<IMAP_DATA_DIR>/
  imap/
    <instrument>/
      <data_level>/
        <year>/
          <month>/
            <filename>for example, withIMAP_DATA_DIR=/data:/data/
  imap/
    swe/
      l0/
        2024/
          01/
            imap_swe_l0_sci_20240105_20240105_v00-00.pktsData Access URLTo change the default URL that the package accesses, you can set
the environment variableIMAP_DATA_ACCESS_URLor within the
packageimap_data_access.config["DATA_ACCESS_URL"]. The default
is the development serverhttps://api.dev.imap-mission.com.TroubleshootingNetwork issuesSSLIf you encounter SSL errors similar to the following:urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>That generally means the Python environment you're using is not finding your system's root
certificates properly. This means you need to tell Python how to find those certificates
with the following potential solutions.Upgrade the certifi packagepipinstall--upgradecertifiInstall system certificatesDepending on the Python version you installed the program with the command will look something like this:/Applications/Python\3.10/Install\Certificates.commandHTTP Error 502: Bad GatewayThis could mean that the service is temporarily down. If you
continue to encounter this, reach out to the IMAP SDC atimap-sdc@lasp.colorado.edu.FileNotFoundErrorThis could mean that the local data directory is not set
up with the same paths as the SDC. See thedata directorysection for an example of how to set this up.File ValidationThis package validates filenames and paths to check they follow our standards, as defined by the filename conventions. There is also a class available for
use by other packages to create filepaths and filenames that follow the IMAP SDC conventions.To use this class, useimap_data_access.ScienceFilepath.Usage:science_file=imap_data_access.ScienceFilePath("imap_swe_l0_sci_20240105_20240105_v00-05.pkts")# Filepath = /imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-05.pktsfilepath=science_file.construct_file_path()