Package: diterator
Iterator through IATI activities from D-PortalThis library makes it easy to downloadIATIactivities fromD-Portaland iterate through them. It takes care of issues like paging, XML parsing, etc. There are wrapper classes for the most-common properties of anIATI activity, and the entire XML is available throughXPathqueries (if desired).UsageThediterator.iterator.Iteratorclass (also available from the package root) creates a stream of IATI activities from D-Portal matching the query parameters you provide:from diterator import Iterator

activities = Iterator({
    "country_code": "ye",
})

for activity in activities:
    print("Activity", activity.identifier)By default, the Iterator class filters duplicate activities out of the results. If you don't want that behaviour, add the parameterdeduplicate=FalseEach activity appears inside awrapperthat gives you easy access to the most-common properties (see documentation below). You can also useXPathqueries to pull out specific pieces of information.D-Portal query parametersFull documentation of the D-Portal query parameters is available here:https://github.com/devinit/D-Portal/blob/master/documents/dstore_q.mdThe most common D-Portal query parameters include the following:ParameterDescriptionExamplereportingThe reporting org nameWorld Health Organizationreporting_codeThe reporting org IATI codeXM-DAC-928aidIATI identifier for a specific activityXM-DAC-928-ER-2016-17-02.006.AF01.ERI01sector_codeA specific OECD-DAC sector code72050titleThe activity title in English.Technical assistance to enable (...)descriptionThe activity description in English.This activity will (...)country_codeTheISO 3166:1-alpha2 codefor a recipient country.SOlocation_nameA name of a specific location (as specified by the publisher).Tombaliday_startThe activity's start date2020-01-01day_endThe activity's end date2020-12-31status_codeThe activity'sstatus code2Use "|" to separate multiple values, and append "_nteq" to the property name for "not equal", "_gt" for "greater than", "_gteq" for "greater than or equal", "_lt" for "less than", "_lteq" for less than or equal, "_glob" for case-sensitive string matching with "*" and "_" as wildcards, or "_like" for case-insensitive string matching.For example, if you wanted to find all activities for Somalia or Kenya in 2020, you could pass this query to the iterator:from diterator import Iterator

activities = Iterator({
    "country_code": "SO|KE",
    "day_end_gteq": "2020-01-01",
    "day_start_lteq": "2020-12-31",
})If you wanted to further filter to only activities with a DAC sector code starting with "7", you could useactivities = Iterator({
    "country_code": "SO|KE",
    "day_end_gteq": "2020-01-01",
    "day_start_lteq": "2020-12-31",
    "sector_code_glob": "7????",
})Again, go to theD-Portal Q API documentationfor full details.Using an XML file or URL instead of the D-Portal Q APIIf you prefer to read IATI activities directly from a URL (e.g. one found at theIATI Registry) or a local file, then you can use the XMLIterator class instead:from diterator import XMLIterator

activities = XMLIterator(url="https://static.rijksoverheid.nl/bz/IATIACTIVITIES20202021.xml")
for activity in activities:
    print(activity.identifier)If your source is a URL rather than a file object or local file name, as in the example above, then you need to put "url=" before it. For a local XML file, leave the "url=" out, so the above example would look like this:from diterator import XMLIterator

activities = diterator.XMLIterator("IATIACTIVITIES20202021.xml")
for activity in activities:
    print(activity.identifier)Wrapper objectsEach activity will be returned wrapped in the Python class diterator.wrappers.Activity, which provides simple access to many of the common properties. All wrappers support some common properties for direct queries against the XML (in case the information you need isn't covered by the properties):MethodDescriptionExampleget_text(xpath_expr)Get the text content of thefirstDOMnode matching theXPathexpression.activity.get_text("reporting-org/@ref")get_narrative(xpath_expr)Get the multilingual narrative content inside the first node matching the XPath expression.activity.get_narrative("title")get_organisation(xpath_expr)Get the organisation info for the first node matching the XPath expression.transaction.get_organisation("provider-org")get_node(xpath_expr)Get the first first DOM node matching the XPath expression.transaction.get_node("value")get_nodes(xpath_expr)Get a list ofallthe DOM nodes matching the XPath expression.activity.get_nodes("sector[@vocabulary=10]")extract_node_text(node)Extract the text from an element or attribute DOM node.Activity objectThis represents the top level of an IATI activity. Available properties:PropertyDescriptionReturn valuedefault_currencyThe activity's default ISO 4217 currency code.stringdefault_languageThe default ISO 639 language code for text content in the activity.stringhumanitarian"Is humanitarian" flag at the activity level.boolean or None if unspecifiedhierarchyActivity hierarchy.Stringlinked_data_uriA URL for a linked-data version of this information.Stringbudget_not_providedAcodeexplaining why there's no budget provided for this activity.StringidentifierThe unique IATI identifier for the activity.Stringreporting_orgThe reporting organisation.Organisationsecondary_reporterIf true, the reporting org is not involved in the activity.boolean or None if unspecifiedtitleThe activity title, possibly in multiple languages.NarrativeTextdescriptionThe activity description, possibly in multiple languages.NarrativeTextparticipating_orgsAll participating organisations.list ofOrganisationparticipating_orgs_by_roleParticipating organisations grouped byrole code.dict with lists ofOrganisationparticipating_orgs_by_typeParticipating organisations grouped bytype code.dict with lists ofOrganisationother_identifiersList of non-IATI alternative activity identifierslist ofIdentifieractivity_statusA code describing thestatus of the activity.stringis_activeConvenience method to show if the activity is currently active.booleanactivity_datesActivity dates grouped by thedate-type code.dict with stringsstart_date_plannedThe planned start date in ISO 8601 format, if specified.stringstart_date_actualThe actual start date in ISO 8601 format, if specified.stringend_date_plannedThe planned end date in ISO 8601 format, if specified.stringend_date_actualThe actual end date in ISO 8601 format, if specified.stringcontact_infoNot yet implementedactivity_scopeAgeographical-scope codefor the activity, if available.stringrecipient_countriesA list of recipient countries.list ofCodedItemrecipient_regionsA list of recipient regions.list ofCodedItemlocationsA list of specific project locations.list ofLocationlocations_by_classSpecific project locations, grouped byclass code.dict with lists ofLocationsectorsA list of project sectors (all vocabularies).list ofCodedItemsectors_by_vocabularySectors grouped bysector vocabulary code.dict with lists ofCodedItemtagsActivity tags (all vocabularies).list ofCodedItemtags_by_vocabularyActivity tags grouped bytag vocabulary code.dict with lists ofCodedItemhumanitarian_scopesHumanitarian scopes (all types and vocabularies).list ofCodedItemhumanitarian_scopes_by_typeHumanitarian scopes grouped bytype code.dict with lists ofCodedItemhumanitarian_scopes_by_vocabularyHumanitarian scopes grouped byvocabulary code.dict with lists ofCodedItempolicy_markersAll policy markers for the activity.list ofCodedItempolicy_markers_by_significancePolicy markers grouped bysignificance codedict with lists ofCodedItempolicy_markers_by_vocabularyPolicy markers grouped byvocabulary codedict with lists ofCodedItemcollaboration_typeCode for thecollaboration type, if specified.stringdefault_flow_typeCode for the defaultflow type, if specified.stringdefault_finance_typeCode for the defaultfinance type, if specified.stringdefault_aid_typesList of defaultaid type codeslist ofCodedItemdefault_aid_types_by_vocabularyDefault aid types grouped byvocabulary codedict withCodedItemdefault_tied_statusCode for the defaulttied statusstringbudgetNot yet implementedplanned_disbursementNot yet implementedcapital_spendNot yet implementedtransactionsAll transactions associated with the activity.list ofTransactionstransactions_by_typeTransactions grouped by theirtype code.dict withTransactiondocument_linkNot yet implementedrelated_activitiesAll related activities.list ofIdentifier(.ref and .type properties used)related_activities_by_typeRelated activities grouped by theirtype code.dict withIdentifier(.ref and .type properties used)legacy_dataNot yet implementedconditionsNot yet implementedresultNot yet implementedcrs_addNot yet implementedfssNot yet implementedTransaction objectRepresents a transaction inside an IATI activity.The class will return default values from the activity if they are not specified for the transaction. If you want to see if the transaction actually specifies the properties itself, use the get_node or get_text methods. For example, the following will specify whether thehumanitarianattribute, sectors, and aid type actually appear on the transaction itself:transaction_humanitarian_flag = transaction.get_text("@humanitarian")
tranaction_has_own_sectors = (len(transaction.get_nodes("sector")) > 0)
transaction_has_own_aid_type = (transaction.get_node("aid-type") is not None)PropertyDescriptionReturn valueactivityThe parent activity.ActivityrefThe transaction reference, if available.stringhumanitarian"Is humanitarian" flag at the transaction level (ignoring the activity level).boolean or None if unspecifieddateTransaction date in ISO 8601 format.stringtypeType code for the transaction.stringvalueTransaction value in its currency (may be negative).floatcurrencyISO 4217 currency code for the transaction (overrides activity default).stringvalue_dateDate to use for currency conversion, in ISO 8601 format.stringdescriptionDescriptive text for the transaction, possibly in multiple languages.NarrativeTextprovider_orgThe source of the funds in the transaction.Organisationreceiver_orgThe destination of the funds in the transaction.Organisationdisbursement_channelA code for the transaction'sdisbursment channel.stringsectorsA list of transaction sectors (all vocabularies), overriding the activity defaults. If the transaction has no sectors, return the activity's sectors as a default.list ofCodedItemsectors_by_vocabularyTransaction sectors grouped bysector vocabulary code. Will default to the activity's sectors if the transaction does not specify its own.dict with lists ofCodedItemrecipient_countriesA list of recipient countries, overriding the activity defaults. If the transaction has no recipient countries, return the activity's recipient countries as a default.list ofCodedItemrecipient_regionsA list of recipient regions, overriding the activity defaults. If the transaction has no recipient regions, return the activity's recipient regions as a default.list ofCodedItemflow_typeA code for the transaction'sflow type. If the transaction has no flow type, return the activity's default flow type.stringfinance_typeA code for the transaction'sfinance type. If the transaction has no finance type, return the activity's default finance type.stringaid_typesA list ofaid type codesspecified for the transaction. If the transaction has no aid types, return the activity's default aid types.list ofCodedItemaid_types_by_vocabularyA dict ofaid type codesspecified for the transaction, keyed by vocabulary. If the transaction has no aid types, return the activity's default aid types.list ofCodedItemtied_statusA code for the transaction'stied status. If the transaction has no tied status, return the activity's default tied status.stringNarrativeText objectRepresents multilingual text wherever it is allowed in IATI. If the language of a translation is unspecified, it will default to the parent activity's default_language property. If you us this object in a string context, it will return the text in the activity's default language (if available) or the first translation specified, so if you wrap this in str(), use it in a print() function, etc, you'll get a simple string.PropertyDescriptionReturn valueactivityThe parent activity.ActivitynarrativesAll available translations, keyed by language.dict of stringOrganisation objectRepresents an organisation (in any context) inside an IATI activity. Some properties will always return None, depending on context.PropertyDescriptionReturn valueactivityThe parent activity.ActivityrefThe organisation identifier, if available.stringtypeThe organisation's type code.stringroleThe organisation's role code (if relevant).stringactivity_idThe organisation's own IATI identifier for the activity (if relevant).stringnameAll translations of the organisation's name.NarrativeTextLocation objectWrapper for a specific location associated with an activity.PropertyDescriptionReturn valueactivityThe parent activity.ActivityrefThe location's code, if available.stringlocation_reachThe location's reach code, if available.stringlocation_idsIdentifiers provided for the location, all vocabularies.list ofCodedItemnameAll translations of the location's name.NarrativeTextdescriptionAll translations of the location's description.NarrativeTextactivity_descriptionTranslations of the description of the activity as it applies in this location.NarrativeTextadministrativesAdministrative codes for this location.list ofCodedItempointLatitude and longitude, space separated.stringpoint_reference_systemName of the lat/lon reference system used.stringexactnessCode for the location exactness.stringlocation_classCode for the location class.stringfeature_designationCode for the location's feature designation.stringCodedItem objectThis class applies to any item that can have a code and optionally, a vocabulary and/or descriptive text. In a string context (e.g. print() or str()), this object will return its code property. Some properties will always return None, depending on context.PropertyDescriptionReturn valueactivityThe parent activity.ActivitycodeCode for this item.stringvocabularyCode for the (relevant) vocabulary in use.stringnarrativeMultiple translations of this item's descriptive text.NarrativeTextpercentageThe percentage applicable to this item within its context and vocabulary, if relevant.stringtypeCode for the type of the item, if relevant.stringlevelCode for the level of the item, if relevant.stringsignificanceCode for the significance of the item, if relevant.stringIdentifier objectRepresents a non-IATI identifier (e.g. iati-activity/other-identifier).PropertyDescriptionReturn valueactivityThe parent activity.ActivityrefThe alternative identifier.stringtypeThe identifiertype code.stringowner_orgThe organisation that owns the identifier, if specified.OrganisationExample# Print the identifier and title of every activity for Somalia from 2019 to 2021

from diterator import Iterator

activities = Iterator({
    "country_code": "so",
    "day_end_gteq": "2019-01-01",
    "day_start_lteq": "2021-12-31",
})

for activity in activities:
    print(activity.identifier, activity.title)You can also get at anything in an activity via the DOM and XPath:transaction_nodes = activity.get_nodes("transaction")

identifier = activity.get_text("iati-identifier")Though normally, to iterate through transactions, you'd just do something likefor transaction in activity.transactions:
    print(transaction.type, transaction.date, transaction.currency, transaction.value)There are also helper properties to group sectors, participating organisations, etcfor sector in activity.sectors_by_vocabulary.get("10", []):
    print(sector.code, sector.percentage, sector.name)AuthorDavid MegginsonLicenseThis code is released into the Public Domain and comes with no warranty of any kind. Use at your own risk.