Package: castoredc-api
CastorEDC APIFeaturesSupports CastorEDC Release 2023.2This is a Python package for interacting with the API of Castor Electronic Data Capture (EDC).
The package contains functions to interact with all the endpoints defined onhttps://data.castoredc.com/api#/.
Within the package are functions for easy export and import of your data through the API.ExportSupported export formats arePandasCSVR (using Feather)ImportImport currently only supports .xlsx files with some configuration.See for more information below.Getting StartedInstall the packagepip install castoredc-apiconda install -c conda-forge castoredc_apiconda install -c reiniervl castoredc_apiImport the clientInstantiate the client with your client-ID and client-secret (don't share these!) and url to the server.ID and secret: Account -> Settings -> Castor EDC APIurl:region.castoredc.comLink the client to your study with the study-IDID: Study -> Settings -> Castor Study IDUse the wrapper functions to start working with your study.For all implemented functions, see:https://data.castoredc.com/api#/fromcastoredc_apiimportCastorClient# Create a client with your credentialsc=CastorClient('MYCLIENTID','MYCLIENTSECRET','data.castoredc.com')# Link the client to your study in the Castor EDC databasec.link_study('MYSTUDYID')# Then you can interact with the API# Get all recordsc.all_records()# Create a new survey packagec.create_survey_package_instance(survey_package_id="FAKESURVEY-PACKAGE-ID",record_id="TEST-RECORD",email_address="obviously@fakeemail.com",auto_send=True)ExportInstantiate the CastorStudy with your credentials, study ID and server url.Use the Study functions to start working with your databaseFor exporting data: The endpoint that extracts data for the study can't be used if the authenticated user has a role within the study.See:https://data.castoredc.com/api#/export/get_study__study_id__export_datafromcastoredc_apiimportCastorStudy# Instantiate Studystudy=CastorStudy('MYCLIENTID','MYCLIENTSECRET','MYSTUDYID','data.castoredc.com')# Export your study to pandas dataframes or CSV filesstudy.export_to_dataframe()study.export_to_csv()# Data and structure mapping are automatically done on export, but you can also map these without exporting your data# Map your study data locally (also maps structure)study.map_data()# Map only your study structure locallystudy.map_structure()# After mapping data and/or structure, you can start working with your study# Get all reportsstudy.get_all_report_forms()# Get all data points of a single recordstudy.get_single_record('000011').get_all_data_points()Data FormattingDate fields are returned as strings (dd-mm-yyyy)Datetime fields are returned as strings (dd-mm-yyyy hh-mm)Numeric fields are all returned as floats.This can be changed by supplying the argument format_options when intialising the CastorStudy.Allowed options are date, datetime, datetime_seconds and time.Seehttps://docs.python.org/3/library/datetime.html#strftime-strptime-behaviorfor formatting options.fromcastoredc_apiimportCastorStudy# Instantiate Study with different formatting settingsstudy=CastorStudy('MYCLIENTID','MYCLIENTSECRET','MYSTUDYID','data.castoredc.com',format_options={"date":"%B%e%Y","datetime":"%B%e%Y %I:%M %p","datetime_seconds":"%B%e%Y %I:%M:%S %p","time":"%I:%M %p",})Missing DataMissing data is mostly handled through pandas (NaN).User-defined missing data is handled through its definitions in Castor.For numeric and text-like variables, these values are -95, -96, -97, -98 and -99.For datetime data, missing data values are with the years 2995, 2996, 2997, 2998, and 2999.ImportInstantiate the CastorStudy with your credentials, study ID and server url.Format your data in the right format (see below)Create a link file to link external and Castor variables (see below)(Optional) Create a variable translation file to translate values and labels to Castor optiongroups (see below).(Optional) Create a merge file to merge multiple columns into one CastorField (see below).(Optional) Set date, datetime and time formatting to translate local format into Castor format (see below).Import your data with the import_data function.If label_data is set to true, it translates the string values to their integer values of the optiongroup in Castor.If set to false, it takes the integer values as is.Data is validated against the Castor database, meaning that:Existence of records and fields is checkedNumeric values are compared against allowed values (min & max)Date(time) and time formats are compared against the specified formatSynchronous UploadThe synchronous upload option uploads each row one by one.When an Error is encountered or the upload finishes successfully, the program outputs the upload log to the output folder and stops.Asynchronous UploadThe asynchronous upload option uploads each row one by one.This is about 15-30 times faster than synchronous upload.The program does not stop if uploading a row encounters an error.When the upload finishes, the program outputs the upload log to the output folder and stops.Error messages are stored in the output folder for debugging.Simple Examplefromcastoredc_apiimportCastorStudyfromcastoredc_apiimportimport_data# Create a Study with your credentialsstudy=CastorStudy('MYCLIENTID','MYCLIENTSECRET','MYSTUDYID','data.castoredc.com')# Import labelled study dataimported_data=import_data(data_source_path="PATH/TO/YOUR/LABELLED/STUDY/DATA",column_link_path="PATH/TO/YOUR/LINK/FILE",study=study,label_data=True,target="Study")# Import labelled study data (asynchronous)imported_data=import_data(data_source_path="PATH/TO/YOUR/LABELLED/STUDY/DATA",column_link_path="PATH/TO/YOUR/LINK/FILE",study=study,label_data=True,target="Study",use_async=True)# Import non-labelled report dataimported_data=import_data(data_source_path="PATH/TO/YOUR/REPORT/DATA",column_link_path="PATH/TO/YOUR/LINK/FILE",study=study,label_data=False,target="Report",target_name="Medication")# Import labelled survey dataimported_data=import_data(data_source_path="PATH/TO/YOUR/LABELLED/SURVEY/DATA",column_link_path="PATH/TO/YOUR/LINK/FILE",study=study,label_data=True,target="Survey",target_name="My first survey package",email="python_wrapper@you-spam.com")Specifying the data structureData filesSee below and example_files/ for an examples.Dates should be formatted as dd-mm-yyyy.Datetime should be formatted as dd-mm-yyyy;hh:mmUse semicolons for fields that allow multiple options (e.g. checkboxes)If any value for a column is translated, all values should have a translation mapped (see below)LabelsThe mg/4 weeks and mg/8 weeks under units will be imported to the med_other_unit fields as they do not match any option of the optiongroup, see link files.Examplepatientmedicationstartdatestopdatedoseunits110001Azathioprine05-12-201905-12-20200.05g/day110002Vedolizumab17-08-201817-09-2020300mg/4 weeks110003Ustekinumab19-12-201703-06-201990mg/8 weeks110004Thioguanine25-04-202027-05-202115mg/day110005Tofacitinib01-03-202031-12-299910mg/dayValuesThe non-integer variables under units will be imported to the med_other_unit fields as they do not match any optionvalue of the optiongroup, see link files.Examplepatientmedicationstartdatestopdatedoseunits110001Azathioprine05-12-201905-12-20200.053110002Vedolizumab17-08-201817-09-2020300mg/4 weeks110003Ustekinumab19-12-201703-06-201990mg/8 weeks110004Thioguanine25-04-202027-05-2021152110005Tofacitinib01-03-202031-12-2999102Link filesLink files should be of the format as shown below.
The mapping is variable name in the Excel file -> variable name in Castor.
If a variable in other is referenced twice in the Castor column, it means that it has a dependency in Castor.This is a way to import data that has an "other" category, for example a radio question that reads:ABCOtherIn which case selecting other opens a new text box to enter this information.
The second variable in the link_file should be this new text box.This is treated in the following manner:First, the data is mapped to the first variable referencedFor all data that could not be mapped to the first variable, the 'other' category is selected in the first variableThen the data that could not be mapped is written to the second variable referenced.Exampleothercastorpatientrecord_idmedicationmed_namestartdatemed_startstopdatemed_stopdosemed_doseunitsmed_unitsunitsmed_other_unitTranslation filesTranslation files link the optiongroup value or label from the external database to the optiongroups from Castor.
Values are translated for all variables specified in the first column of the file.Two situations can occur when a value is encountered for which no translation is given:If a dependent field is specified (see link files): the value is not translated and imported to the dependent field.If no dependent field is specified: the program gives an error. In this situation, every value that occurs in the external database needs to be mapped.Examplevariableothercastorfamily disease historynoneNonefamily disease historydon't knowUnknownfamily disease historydeafDeafnessfamily disease historycardiomyopathy(Cardio)myopathyfamily disease historyencephalopathyEncephalopathyfamily disease historydiabetesDiabetes Mellitusfamily disease historycardiovascular diseaseHypertension/Cardiovascular diseasefamily disease historythromboembolismThrombosisfamily disease historytumorMalignancyfromcastoredc_apiimportCastorStudyfromcastoredc_apiimportimport_data# Create a Study with your credentialsstudy=CastorStudy('MYCLIENTID','MYCLIENTSECRET','MYSTUDYID','data.castoredc.com')# Import study data with a translation fileimported_data=import_data(data_source_path="PATH/TO/YOUR/LABELLED/STUDY/DATA",column_link_path="PATH/TO/YOUR/LINK/FILE",study=study,label_data=True,target="Study",translation_path="PATH/TO/YOUR/TRANSLATION/FILE")Merge filesMerge files link the multiple columns from the external database to a single checkbox field in Castor.For each column from the external database specified under other_variable the value under other_value is mapped to the castor_value for the castor_variable.If specifying a merge file, note that castor_value is the new other variable for your link file (see below).All other_values not defined raise an Error.
Only supports many-to-one matching.ExampleData Filepatientdate baseline blood samplebaseline hemoglobinfactor V Leidendatetime onset stroketime onset trombectomyyear of birthpatient sexpatient racefamhist_nonefamhist_deaffamhist_cardiomyopathyfamhist_encephalopathyfamhist_diabmellfamhist_cardiovascfamhist_malignancyfamhist_unknown11000116-03-20218.355;16-03-202116-03-2021;07:3009:251999FemaleAsianNoNoYesYesYesNoNoNo11000217-03-20217.233;17-03-202117-03-2021;15:3006:331956FemaleAfrican/blackNoYesYesNoNoNoNoNo11000316-03-20229.1-45;18-03-202218-03-2022;02:0012:241945MaleChineseYesNoNoNoNoNoNoNo11000417-03-20223.228;19-03-202217-03-2022;21:4323:231933MaleCaucasian/whiteNoNoNoNoNoYesYesNo11000516-03-202310.35;20-03-202316-03-2023;07:2208:141921FemaleHispanicNoNoNoNoNoNoNoYesMerge Fileother_variableother_valuecastor_variablecastor_valuefamhist_noneYeshis_familyNonefamhist_deafYeshis_familyDeafnessfamhist_cardiomyopathyYeshis_family(Cardio)myopathyfamhist_encephalopathyYeshis_familyEncephalopathyfamhist_diabmellYeshis_familyDiabetes Mellitusfamhist_cardiovascYeshis_familyHypertension/Cardiovascular diseasefamhist_malignancyYeshis_familyMalignancyfamhist_unknownYeshis_familyUnknownfamhist_noneNohis_familyfamhist_deafNohis_familyfamhist_cardiomyopathyNohis_familyfamhist_encephalopathyNohis_familyfamhist_diabmellNohis_familyfamhist_cardiovascNohis_familyfamhist_malignancyNohis_familyfamhist_unknownNohis_familyLink Fileothercastorpatientrecord_iddate baseline blood samplebase_bl_datebaseline hemoglobinbase_hbfactor V Leidenfac_V_leidendatetime onset strokeonset_stroketime onset trombectomyonset_trombectomyyear of birthpat_birth_yearpatient sexpat_sexpatient racepat_racehis_familyhis_familyfromcastoredc_apiimportCastorStudyfromcastoredc_apiimportimport_data# Create a Study with your credentialsstudy=CastorStudy('MYCLIENTID','MYCLIENTSECRET','MYSTUDYID','data.castoredc.com')# Import study data with a merge fileimported_data=import_data(data_source_path="PATH/TO/YOUR/LABELLED/STUDY/DATA",column_link_path="PATH/TO/YOUR/LINK/FILE",study=study,label_data=True,target="Study",merge_path="PATH/TO/YOUR/MERGE/FILE")Data FormattingStandard date formatting settings are the following.
Date(time) and time fields should follow these formats in the Excel sheet to be uploaded.Date = dd-mm-yyyyDatetime = dd-mm-yyyy;hh-mmTime = hh:mmDecimal separator = .These can be changed by supplying the argument format_options when calling create upload.Allowed options are date, datetime, and time. Decimal separator cannot be changed.Seehttps://docs.python.org/3/library/datetime.html#strftime-strptime-behaviorfor formatting options.fromcastoredc_apiimportCastorStudyfromcastoredc_apiimportimport_data# Create a Study with your credentialsstudy=CastorStudy('MYCLIENTID','MYCLIENTSECRET','MYSTUDYID','data.castoredc.com')# Import labelled study data with changed formatsimported_data=import_data(data_source_path="PATH/TO/YOUR/LABELLED/STUDY/DATA",column_link_path="PATH/TO/YOUR/LINK/FILE",study=study,label_data=True,target="Study",format_options={"date":"%B%d%Y","datetime":"%B%d%Y %I:%M %p","time":"%I:%M %p",})PrerequisitesPython Version >= 3.8Seerequirements.txtKnown IssuesAsync import and export cannot be used in an IPython environment, seethis discussionFeather export is uncompressed, seethis issueDevice token and Econsent endpoints are untested. Use at your own risk.ContributingPlease readCONTRIBUTING.mdfor details on our code of conduct, and the process for submitting pull requests to us.Running TestsWant to contribute to the testing suite? Or test possible changes you want to contribute?
Tests can be ran via two methods: on Github and on your local machine.On Github: when you create a pull request for this project, Pytest automatically runs for the testing suite (seepytest.yml).
If you have added a whole new testing module, don't forget to add this to the pytest.yml file. Within the repository, applicable access rights have been set for the client.
Use the following fixtures for the respective modules you want to test:Testing API read endpoints: clientTesting API write endpoints: write_clientTesting output:For structure output to Python: integration_study & integration_study_formatFor data output to Python: integration_study_mapped & integration_study_format_mappedFor data output to csv: output_dataTesting importing of data: import_studyLocally: You can only run tests locally when you have read and write access to the correct Castor Studies.
Please send a message to the repository owner to ask for the correct access, with information on why.
Access and correct study IDs will then be given to run the tests.VersioningWe useSemVerfor versioning. For the versions available, see thetags on this repository.AuthorsR.C.A. van Linschoten-Initial Development-Reinier van LinschotenSee also the list ofcontributorswho participated in this project.LicenseThis project is licensed under the MIT License - see theLICENSE.mdfile for detailsAcknowledgmentsFranciscus Gasthuis & Vlietland for making time available for developmentCastor EDC for support and code review