Package: riptide-ffa
riptideriptide("searching forpulsars in thetimedomain") is a pulsar searching package implementing the Fast Folding Algorithm (FFA), the theoretically optimal search method for periodic signals. Its interface is entirely in python while the core algorithms are implemented in C. riptide provides:A library of functions and classes to use interactively to process dedispersed time seriesA pipeline executable to process a set of DM trials and output a list of candidate files and other useful data productsCitationIf usingriptidecontributes to a project that leads to a scientific publication, please cite the article:"Optimal periodicity searching: Revisiting the Fast Folding Algorithm for large scale pulsar surveys"Sensitivity of the FFAThe article covers the topic of the FFA's sensitivity in theory, here is a practical example on a faint source. Below is an L-band observation ofPSR J1932-3655from theSUPERB survey, significantly offset from the true position of the source. The first plot was obtained by folding the observation using the known ephemeris of the pulsar withPSRCHIVE. This is one of the faintest known pulsar instances identifiable in the survey.And below is a blind detection of the pulsar, runningriptideon the same observation:DocumentationA quickstart guide, a pipeline configuration tutorial and the full API documentation can be foundhere.InstallationThe easiest method is to use pip install, which pulls the latest release from the python package index and installs all required dependencies:pip install riptide-ffaThe alternative is to clone the repository, especially if you want the absolute latest version:gitclonehttps://github.com/v-morello/riptideAnd then in the base directory ofriptiderunmakeinstallThis simply runspip installineditable mode, which means you can freely edit the code. It also installs any required dependencies withpipthat are not present already. The installer also adds links to two command-line apps in your python environment usingconsole_scripts entry points:rffa: The full end-to-end pipeline to search multiple DM trials, see below for details on how to use it.rseek: A lightweight app to search a single time series and print significant candidates found, useful for quick data checks.You may check that it all works by typing:rffa -h
rseek -hAnd you should see the full help of both applications.ChangelogAll notable changes to this project will be documented in this file.The format is based onKeep a Changelog,
and this project adheres toSemantic Versioning.0.2.5 - 2023-12-23This version fixes an installation error on recent OSX versions, and a segmentation fault that triggers in some edge cases.Thanks to Francesco Coti Zelati forreporting the original issuethat led to both fixes.FixedFixed a serious bug where some intermediate output arrays were unexpectedly allocated with a size one element smaller than expected, causing segmentation faults. This was found to be a side effect of the compilation flag-freciprocal-math. See issue#4for a detailed explanation. The set of compilation flags for the C++ extensions has been reviewed and adjusted.The C++ extensions can now be built with the C++17 standard enabled without throwing any errors or warnings. Recentclangversions (the default compiler on OSX) would throw a number of narrowing conversion errors, which would in turn make the installation of riptide fail with a cryptic error message.0.2.4 - 2022-02-03This version fixes a bug where all the functions in the python bindings to the C++ code (exposed inriptide.libcppon the Python side) assume that their input numpy arrays are contiguous in memory. In practice, passing a column slice of a two-dimensionalfloat32array tofast_running_medianwas found to produce incorrect results: the rather than readingdata[:, col_index], the code readdata[0, col_index:col_index+num_cols]. The issue would only trigger on thefloat32type, because otherwise an implicit copy (contiguous in memory) of the input was created.Thanks to Akshay Suresh for finding the bug and reporting it.FixedAll C++ functions inpython_bindings.cppnow explicitly check that all input arrays are contiguous in memory, and throwstd::invalid_argumentotherwise (maps toValueErrorin Python).The size equality check of the input arrays for therollbackfunction inpython_bindings.cppis now correctAddedrunning_medianfunction that wraps its counterpart in the C++ library. It ensures that the array passed to it is contiguous in memory, and if not, makes a temporary contiguous copy before doing so.Unit test that checks the correctness ofrunning_medianon non-contiguous data slices.Theriptidemodule now exposes the functionsrunning_medianandfast_running_median. Document how boundary conditions are handled in running median calculation. Added entries for both functions in the documentation, in Kernel Functions section.ChangedRenamed the filerunning_median.pytorunning_medians.pyto avoid name collisions.0.2.3 - 2021-08-01UpdatedPackaging is now PEP 517/518 compliantUsingsetuptools-scminstead ofversioneer(which is not maintained anymore) for automatic versioning0.2.2 - 2021-07-24AddedAdded full documentation and readthedocs integration. The docs can be foundhere.Fixed typos in docstrings, extra docstrings forPeriodogramclassUpdatedBase Dockerfile on python:3.8, ensure all tests pass during the buildFixedIt is now possible to specify period ranges so that the data are searched at their raw resolution (no downsampling). This corresponds to choosing period range parameters such thatperiod_min = bins_min x tsamp. In this case, the code would previously raise an error saying that it could not downsample the data with a factor equal to 1, which was not the intended behaviour.Fixed an issue where the C++ functionscheck_trial_widthsandcheck_stdnoisewould systematically throw an exception regardless of their input arguments. The issue was occurring withgcc 8.3.0on some systems, and appears to have been caused by said functions having a return typeboolwhile they actually did not return anything. The functions havevoidreturn type now.0.2.1 - 2020-10-27AddedLimited support for X-ray and Gamma time series in PRESTO inf/dat format.TimeSeries.from_presto_inf()will now correctly read these. However, a warning will be issued when doing so, becauseriptideis currently designed to process data where the background noise is Gaussian.Be aware that processing high-energy data may produce junk results.FixedDocstring ofTimeSeries.generate()was missing theperiodparameter0.2.0 - 2020-08-10This release contains significant improvements and additions but breaks compatibility withv0.1.x.ChangedTheffa_search()function now only returns two values: the de-reddenedTimeSeriesthat was actually searched, and aPeriodogramobject. TheProcessingPlanclass has been removed.Clean rewrite of all kernels in C++. Python bindings for said kernels are now generated withpybind11which requires a lot less boilerplate code. Kernel functions can imported in python from theriptide.libcppsubmodule.The python functionget_snr()has been renamedboxcar_snr()Moved unit tests toriptide/testsso they can be packaged with the rest of the moduleAddedThePeriodogramreturned byffa_searchnow has afoldbinsmember, which is the number of phase bins that were used when folding the data for a particular trial period. Using this information, the pipeline now returns thetrueduty cycle of the candidates; it was previously returning only an estimate equal towidth / bins_avgwherewidthwas the best pulse width in bins, andbins_avg = (bins_min + bins_max) / 2.running_median()function in C++, it is around 8x faster than its former python counterpart.Tests forrunning_median()Tests forboxcar_snr()DM trial selection by the pipeline is now considerably fasterThe pipeline now infers the observing min/max frequencies and the number of channels from the input data if possible. It is now recommended to leave the fieldsfmin,fmax, andnchansblank in the pipeline configuration file. It is necessary to specify these values only when the dedispersed time series data format (e.g. SIGPROC) does not contain that information.In the pipeline configuration file, the minimum and maximum trial DM fields can also be left blank. If so, the minimum and/or maximum DMs of the search are determined by the DM trials available to process.Added careful validation of the pipeline configuration file (using theschemalibrary). The pipeline will raise an exception early in the processing if the configuration is invalid, with a helpful error message.Can now run the unit test suite from python code or IPython by callingriptide.test()RemovedRemovedProcessingPlanclass. It used to be passed to the old C function that was computing the periodogram. Its job is now directly performed by the new C++ functionperiodogram().Removed old C kernels0.1.5 - 2020-05-23Changedfind_peaks()now always returns peaks in decreasing S/N orderAddedrseekcommand-line application that searches a single dedispersed time series and prints a list of significant peaks found0.1.4 - 2020-05-03This version reduces RAM usage significantly. Trial periods are now stored as double precision floats.FixedTrial periods are now stored asdoubleprecision floats (doublein C,floatin python). When searching very long time series (several hours) and short trial periods, single precision floats were inaccurate enough that small groups of consecutive period trials erroneously ended up having the exact same value. Incorrect detection periods would propagate down the pipeline, andCandidateobjects could end up being folded with a period slightly offset from the true value.ChangedThe pipeline worker sub-processes now get passed afile pathto a time series as an input, rather than a fullTimeSeriesobject, which saves the cost of communicating a lot of data between processesThe buffers used for downsampling and FFA transforming the data when calculating periodograms are now given the smallest possible size. They only need to holdN / fdata points, whereNis the number of samples in the raw time series, andfthe first downsampling factor in the search plan. Buffers were previously given a size ofNwhich was often wasteful.AddedAdded option to save pipeline logs to file0.1.3 - 2020-04-27FixedFixed a rare but serious bug in the C function downsample() where the code would attempt to read one element beyond the end of the input array. This would happen only when the downsampling factor was such thatnsamp_input / dsfactorwas exactly an integer. This would cause the last sample in the output to have an absurd value, or a segmentation fault.Fixed a crash in the pipeline that would occur when significant peak clusters were found, but none had a S/N that would exceed thecandidate_filters.snr_minconfiguration parameter. Turning the resulting empty list of clusters into a CSV would then fail.Fixed a related problem where clusters whose S/N was belowcandidate_filters.snr_minwere not saved to the CSV file, which was not the intended behaviour.AddedUnit test suite, to be improved and expandedTravis CICodecov integration0.1.2 - 2020-04-15FixedMetadatais now correctly carrying the "tobs" attribute for TimeSeries loaded from SIGPROC data. This was causing a cryptic error when processing SIGPROC dedispersed time series with the pipeline.AddedCan now read and process 8-bit SIGPROC dataCan now read SIGPROC header keys of boolean type. In particular the "signed" key (which defines the signedness of 8-bit SIGPROC data) is now supported by default.0.1.1 - 2020-04-08FixedModule should now properly install on OSX, where some C compilation options had to be adapted.numpy.ctypeslibalso expects shared libraries to have a.dylibextension on OSX rather than the linux standard.so0.1.0 - 2020-04-08This release contains major improvements and additions but breaks compatibility withv0.0.x. If you have any ongoing large-scale searches or projects, stick with the older release you were using. Other users shoulddefinitelyuse this new version.FixedEnsure that each subprocess spawned by the pipeline consumes only one CPU, as initially intended to achieve optimal CPU usage. In previous versions, somenumpyfunctions would attempt to run on all CPUs at once which was detrimental. As a result the pipeline is now considerably faster.The impact of downsampling by a non-integer factor on the noise variance is now correctly dealt with when normalizing the output S/N values. Refer to the paper for details. The difference should be negligible, except when downsampling the original input data by factors close to 1.The Makefile used to build the C sources does not explicitly setCC=gccanymore. The default system compiler will now be used instead.gccwill be used by default if the environment variableCCis undefined.ChangedClean rewrite of peak detection algorithm. It uses the same method as before, but the arguments of thefind_peaks()function have changed. See docstring.Now using JSON instead of HDF5 as the data product file format. This is vastly easier to maintain and future-proof. Read/write speed and file sizes remain similar.Clean rewrite of the pipeline, which has been improved and runs faster, see below for all related changes.Improved DM trial selection, now uses a method similar to PRESTO'sDDPlanto achieve the least expensive DM space coverageInput de-reddening and normalisation is now common to all search period sub-ranges, further improving pipeline run timeHarmonic flagging is now always performedRemoving harmonics from the output candidate list is now optionalAdded option to produce all candidate plots at the end of the pipeline runCandidate plots look nicerSaving candidate files and plots is done with multiple CPUs and runs fasterThe pipeline configuration file keywords / schema has been adjusted to match all pipeline changes, see example provided and documentationCandidateclass has been refactored, its attribute and method names have changedChanged name of high level FFA transforms toffa1()andffa2().Updated signal generation functions so that the 'amplitude' parameter now represents the expected true signal-to-noise ratioAddedTimeSeriesnow has afold()method that returns a numpy array representing either sub-integrations or a folded profileTiming of all pipeline stagesDynamic versioning using theversioneermodule. In python, the current version is accessible viariptide.__version__Addedffafreq()andffaprd()to generate list of FFA transform trial freqs / periods.RemovedRemoved theSubIntegrationsclass which added unneeded complexity, sub-integrations are now represented as a 2D numpy array0.0.3 - 2019-11-30AddedFull docstring forffa_search()ChangedCleaner and faster FFA C kernels, which have been moved to separate filesSlight optimisation to S/N calculation, where only the best value across all pulse phases is normalized, instead of normalizing at every phase trialS/N calculation separated into smaller functionsRemoved OpenMP multithreading from S/N calculation, it was slower in most usual cases. The benefits were visible only for very large input data. As a result,ffa_search()does not accept the 'threads' keyword anymore, and the 'threads' keyword has also been removed from the pipeline configuration files (in the 'search' section). Parallelism only happens at the TimeSeries level, i.e. one process per TimeSeries.FixedReinstated the--fast-mathcompilation option that had been accidentally removed in v0.0.2. The code is now much faster.0.0.2 - 2019-11-05Addedriptideis now properly packaged with asetup.pyscript. Updated installation instructions inREADME.mdUpdated Dockerfile. Build docker image withmake dockercommand.FixedWhen normalising TimeSeries, use a float64 accumulator when calculating mean and variance. This avoid saturation issues on data with high values, e.g. from 8-bit Parkes UWL observations.ChangedImproved candidate plots: docstring forCandidateclass, DM unit on plots, option to subtract the baseline value of the profile before displaying it, option to plot the profile as either a bar or line chart.0.0.1 - 2018-10-25AddedFirst stable release of riptide. This is the version that will be run on the LOTAAS survey.