Package: ripe.atlas.sagan
RIPE Atlas SaganA parsing library for RIPE Atlas measurement resultsWhy this existsRIPE Atlas generates alotof data, and the format of that data changes over
time. Often you want to do something simple like fetch the median RTT for each
measurement result between dateXand dateY. Unfortunately, there are
dozens of edge cases to account for while parsing the JSON, like the format of
errors and firmware upgrades that changed the format entirely.To make this easier for our users (and for ourselves), we wrote an easy to use
parser that’s smart enough to figure out the best course of action for each
result, and return to you a useful, native Python object.How to installThe stable version should always be in PyPi, so you can install it withpip:$pipinstallripe.atlas.saganBetter yet, make sure you get ujson and sphinx installed with it:$pipinstallripe.atlas.sagan[fast,doc]TroubleshootingSome setups (like MacOS) have trouble with building the dependencies required
for reading SSL certificates. If you don’t care about SSL stuff and only want to
use sagan to say, parse traceroute or DNS results, then you can do the following:$SAGAN_WITHOUT_SSL=1pipinstallripe.atlas.saganQuickstart: How To Use ItYou can parse a result in a few ways. You can just pass the JSON-encoded string:fromripe.atlas.saganimportPingResultmy_result=PingResult("<result string from RIPE Atlas ping measurement>")print(my_result.rtt_median)123.456print(my_result.af)6You can do the JSON-decoding yourself:fromripe.atlas.saganimportPingResultmy_result=PingResult(json.loads("<result string from RIPE Atlas ping measurement>"))print(my_result.rtt_median)123.456print(my_result.af)6You can let the parser guess the right type for you, though this incurs a small
performance penalty:fromripe.atlas.saganimportResultmy_result=Result.get("<result string from RIPE Atlas ping measurement>")print(my_result.rtt_median)123.456print(my_result.af)6What it supportsEssentially, we tried to support everything. If you pass in a DNS result string,
the parser will return aDNSResultobject, which contains a list ofResponse’s, each with anabufproperty, as well as all of the
information in that abuf: header, question, answer, etc.fromripe.atlas.saganimportDnsResultmy_dns_result=DnsResult("<result string from a RIPE Atlas DNS measurement>")my_dns_result.responses[0].abuf# The entire stringmy_dns_result.responses[0].abuf.header.arcount# Decoded from the abufWe do the same sort of thing for SSL measurements, traceroutes, everything. We
try to save you the effort of sorting through whatever is in the result.Which attributes are supported?Every result type has its own properties, with a few common between all types.Specifically, these attributes exist on all*Resultobjects:createdAn datetime object of thetimestampfieldmeasurement_idprobe_idfirmwareAn integer representing the firmware versionoriginThefromattribute in the resultis_errorSet toTrueif an error was foundAdditionally, each of the result types have their own properties, likepacket_size,responses,certificates, etc. You can take a look at
the classes themselves, or just look at the tests if you’re curious. But to get
you started, here are some examples:# Pingping_result.packets_sent# Intping_result.rtt_median# Float, rounded to 3 decimal placesping_result.rtt_average# Float, rounded to 3 decimal places# Traceroutetraceroute_result.af# 4 or 6traceroute_result.total_hops# Inttraceroute_result.destination_address# An IP address string# DNSdns_result.responses# A list of Response objectsdns_result.responses[0].response_time# Float, rounded to 3 decimal placesdns_result.responses[0].headers# A list of Header objectsdns_result.responses[0].headers[0].nscount# The NSCOUNT value for the first headerdns_result.responses[0].questions# A list of Question objectsdns_result.responses[0].questions[0].type# The TYPE value for the first questiondns_result.responses[0].abuf# The raw, unparsed abuf string# SSL Certificatesssl_result.af# 4 or 6ssl_result.certificates# A list of Certificate objectsssl_result.certificates[0].checksum# The checksum for the first certificate# HTTPhttp_result.af# 4 or 6http_result.uri# A URL stringhttp_result.responses# A list of Response objectshttp_result.responses[0].body_size# The size of the body of the first response# NTPntp_result.af# 4 or 6ntp_result.stratum# Statum idntp_result.version# Version numberntp_result.packets[0].final_timestamp# A float representing a high-precision NTP timestampntp_result.rtt_median# Median value for packets sent & receivedWhat it requiresAs you might have guessed, with all of this magic going on under the hood, there
are a few dependencies:cryptography(Optional: see “Troubleshooting” above)python-dateutilpytzIPyAdditionally, we recommend that you also installujsonas it will speed up
the JSON-decoding step considerably, andsphinxif you intend to build the
documentation files for offline use.Running TestsThere’s a full battery of tests for all measurement types, so if you’ve made
changes and would like to submit a pull request, please run them (and update
them!) before sending your request:$pythonsetup.pytestYou can also installtoxto test everything in all of the supported Python
versions:$pipinstalltox$toxFurther DocumentationComplete documentation can always be found onRead the Docs,
and if you’re not online, the project itself contains adocsdirectory –
everything you should need is in there.Who’s Responsible for This?Sagan is actively maintained by the RIPE NCC and primarily developed byDaniel
Quinn, while the abuf parser is mostly the responsibility ofPhilip Homburgwith an assist from Bert Wijnen and Rene Wilhelm who contributed to the original
script.Andreas Stirkosdid the bulk of the work on NTP measurements and
fixed a few bugs, and big thanks go toChris Amin,John Bond, andPier Carlo Chiodifor finding and fixing stuff where they’ve run into
problems.ColophonBut why “Sagan”? The RIPE Atlas team decided to name all of its modules after
explorers, and what better name for a parser than that of the man who spent
decades reaching out to the public about the wonders of the cosmos?Changelog1.3.1Ping edge case fixAdd py37 testing1.3.0abuf.py: error handling for NS records, extended rcode, cookies and client subnets1.2.2Catch problems parsing SSL certificates1.2.1Add support for non-DNS names in subjectAltName extensions1.2Replaced pyOpenSSL with cryptographyAdded parsing of subjectAltName X509 extension1.1.11Added first version of WiFi results1.1.10Added aparse_all_hopskwarg to the Traceroute class to tell Sagan to stop parsing Hops and Packets once we have all of the last hop statistics (default=True)Remove dependency on IPy: we were using it for IPv6 canonicalization, but all IPv6 addresses in results should be in canonical form to start with.1.1.9Removed theparse_abufscript because no one was using it and its
Python3 support was suspect anyway.1.1.8Handle case where a traceroute result might not havedst_addrfield.1.1.7Change condition of traceroute’slast_hop_respondedflag.Add couple of more traceroute’s properties.is_successandlast_hop_errors.Add tests to the package itself.1.1.6Fix forIssue #56a case where theqbufvalue wasn’t being properly
captured.Fixed small bug that didn’t accurately capture theDOproperty from
the qbuf.1.1.5We now ignore so-called “late” packets in traceroute results.  This will
likely be amended later as future probe firmwares are expected to make
better use of this value, but until then, Sagan will treat these packets
as invalid.1.1.4Added atypeattribute to allResultsubclassesAdded support for a lot of new DNS answer types, includingNSEC,PTR,SRV, and more.  These answers do not yet have a complete
string representation however.1.1.3Changed the name ofTracerouteResult.rtt_mediantoTracerouteResult.last_rtt_median.Modified theDnsResultclass to allow the “bubbling up” of error
statuses.1.1.2We skipped this number for some reason :-/1.1.1Fixed astring representation bugfound byiortiz1.1.0Breaking Change: theAuthorityandAdditionalclasses were
removed, replaced with the appropriate answer types.  For the most part,
this change should be invisible, as the common properties are the same,
but if you were testing code against these class types, you should
consider this a breaking change.Breaking Change: The__str__format for DNSRrsigAnswerto
conform the output of a typicaldigbinary.Added__str__definitions to DNS answer classes for use with the
toolkit.In an effort to make Sagan (along with Cousteau and the toolkit) more
portable, we dropped the requirement for thearrowpackage.1.0.01.0! w00t!Breaking Change: thedataproperty of theTxtAnswerclass was
changed from a string to a list of strings.  This is a correction from
our own past deviation from the RFC, so we thought it best to conform as
part of the move to 1.0.0Fixed a bug where non-ascii characters in DNS TXT answers resulted in an
exception.0.8.2Fixed a bug related to non-ascii characters in SSL certificate data.Added a wrapper for json loaders to handle differences between ujson and
the default json module.0.8.1Minor fix to make allResultobjects properly JSON serialisable.0.8.0Addediortiz’s patch for flags andflagsandsectionsproperties on DNSAnswerobjects.0.7.1ChangedREADME.mdtoREADME.rstto play nice with pypi.0.7Addedpierky’s newRRSigAnswerclass to
the dns parser.0.6.3Fixed a bug in how Sagan deals with inappropriate firmware versions0.6.2Addedpierky’s fix to fix AD and CD flags
parsing in DNS Header0.6.1Addedrtt_min,rtt_max,offset_min, andoffset_maxtoNTPResult0.6.0Support for NTP measurementsFixes for how we calculate median valuesSmarter setup.py0.5.0Complete Python3 support!0.4.0Added better Python3 support.  Tests all pass now for ping, traceroute,
ssl, and http measurements.Modified traceroute results to make use ofdestination_ip_respondedandlast_hop_responded, deprecatingtarget_responded.  See the
docs for details.0.3.0Added support for making use of some of the pre-calculated values in DNS
measurements so you don’t have to parse the abuf if you don’t need it.Fixed a bug in the abuf parser where a variable was being referenced by
never defined.Cleaned up some of the abuf parser to better conform to pep8.0.2.8Fixed a bug where DNSTXTresults with classINwere missing a.datavalue.Fixed a problem in the SSL unit tests where\nwas being
misinterpreted.0.2.7Made abuf more robust in dealing with truncation.0.2.6ReplacedSslResult.get_checksum_chain()with theSslResult.checksum_chainproperty.Added support for catching results with anerrproperty as an actual
error.0.2.5Fixed a bug in how theon_errorandon_malformationpreferences
weren’t being passed down into the subcomponents of the results.0.2.4Support forseconds_since_syncacross all measurement types0.2.3“Treat a missing Type value in a DNS result as a malformation” (Issue #36)0.2.2Minor bugfixes0.2.1Added amedian_rttvalue to tracerouteHopobjects.Smarter and more consistent error handling in traceroute and HTTP
results.Added anerror_messageproperty to all objects that is set toNoneby default.0.2.0Totally reworked error and malformation handling.  We now differentiate
between a result (or portion thereof) being malformed (and therefore
unparsable) and simply containing an error such as a timeout.  Look for
anis_errorproperty or anis_malformedproperty on every object
to check for it, or simply passon_malformation=Result.ACTION_FAILif
you’d prefer things to explode with an exception.  See the documentation
for more detailsAdded lazy-loading features for parsing abuf and qbuf values out of DNS
results.Removed the deprecated properties fromdns.Response.  You must now
access values likeedns0fromdns.Response.abuf.edns0.More edge cases have been found and accommodated.0.1.15Added a bunch of abuf parsing features fromb4ldrwith some help fromphicoh.0.1.14Fixed the deprecation warnings inDnsResultto point to the right
place.0.1.13Better handling ofDNSResulterrorsRearranged the way abufs were handled in theDnsResultclass to make
way forqbufvalues as well.  The old method of accessingheader,answers,questions, etc is still available viaResponse, but
this will go away when we move to 0.2.  Deprecation warnings are in place.0.1.12Smarter code for checking whether the target was reached inTracerouteResults.We now handle thedestination_option_sizeandhop_by_hop_option_sizevalues inTracerouteResult.Extended support for ICMP header info in tracerouteHopclass by
introducing a newIcmpHeaderclass.0.1.8Broader support for SSL checksums.  We now make use ofmd5andsha1, as well as the originalsha256.