Package: viewser
viewserViewser is a software package allowing any user to fetch raw data from the VIEWS database, apply a wide range
of transforms to the raw data, and download the resulting dataset as a single pandas dataframe.
The following documentation refers to viewser v>=6.0.0. This is a major update to viewser. Principal changes from
older versions of viewser aredata is now fetched from the database hosted on new dedicated hardware hosted at PRIOthe API which fetches data has been changed - users no longer specify the table from which raw data is
fetched, only the level of analysis.Installing viewserViewser is a python package and it is strongly advised that it be installed in a conda enviroment. Viewser itself
is not available on conda and can only be installed via pippip install viewserConfigurationThe tool is configured using theviewser config set KEY VALUEandviewser config load JSONcommands. The settings shown without defaults here, or with
defaults that don't make sense for the average user (REMOTE_URL) must be
configured before use.SettingDescriptionDefaultRETRY_FREQUENCYGeneral request retry frequency in seconds5QUERYSET_MAX_RETRIESHow many times a queryset is queried before failing500LOG_LEVELDetermines what logging messages are shownINFOERROR_DUMP_DIRECTORYDetermines where error dumps are written to~/.views/dumpsREMOTE_URLURL of a views 3 instancehttp://0.0.0.0:4000MODEL_METADATA_DATABASE_HOSTNAMEHostname of database for storing model metadatahermesMODEL_METADATA_DATABASE_NAMEDBname of database for storing model metadataforecasts3MODEL_METADATA_DATABASE_USERUsername for database for storing model metadataInferred from certMODEL_METADATA_DATABASE_SSLMODESSLmode for database for storing model metadatarequiredMODEL_METADATA_DATABASE_PORTPort of database for storing model metadata5432MODEL_METADATA_DATABASE_SCHEMASchema of database for storing model metadataforecastsMODEL_METADATA_DATABASE_TABLETable of database for storing model metadatamodelAZURE_BLOB_STORAGE_ACCOUNT_NAMEName of Azure blob storage accountAZURE_BLOB_STORAGE_ACCOUNT_KEYAccess key of Azure blob storage accountUsageThe viewser client can be used in two ways:Via command-line interface (CLI)viewserfunctionality is exposed via a CLI on your system after installation.
An overview of available commands can be seen by runningviewser --help.The CLI is envisaged mainly as a tool to help users with issues such as selecting appropriate transforms, exploring the database, determining the level of analysis of a given feature, etc.Useful commandsShow all tables in the database:viewser tables listShow columns in a particular table:viewser tables show table-nameShow all transforms sorted by level of analysis:viewser transforms listShow docstring for a particular transform:viewser transforms show transform-name(note that the transform name should be the full name, e.g. 'any/temporal.entropy')Via APIThe full functionality of viewser is exposed via its API for use in scripts and notebooksThe two fundemntal objects used to define what data is fetched by the client are theQuerysetand theColumn, where a
Queryset consists of one or more Columns.To define a queryset, one first imports the Queryset and Column classesfrom viewser import Queryset, ColumnA new queryset is then created bynew_queryset = (Queryset("queryset_name", "level of analysis"))Every queryset must havea (meaningful) name.a level of analysis. This defines the level of analysis that all columns in the queryset will be aggregated/disaggregated to, and therefore the level of analysis at which the final dataset will be presented.The level of analysis must be a string of the form spatial-unit_time-unit, where the spatial-unit must be one ofcountrypriogridand the time unit must be one ofmonthyearColumn objects representing data are added to the queryset using thewith_columnmethod:new_queryset = (Queryset("simple_conflict", "country_month")

           .with_column(Column("ged_sb", from_loa="country_month", from_column="ged_sb_best_sum_nokgi")
                        )
            )Each call towith_columntakes a single Column instance as its argument.The Column instance in turn defines what the column is to be called and what data will go into it. The first argument to the Column instance is the column name. This again should be meaningful.Note that, unlike older versions of viewser (<6.0.0), all column names in a querysetmustbe unique. If two or more columns are given the same name, the queryset will be rejected by the server and an error massage detailing which columns have repeated names will be returned.The second argument to the Column instancefrom_loaspecifies which level of analysis the requested raw data is defined at. If the user does not know this, they need to examine the database using the viewser CLI.If the wrong loa is specified, the queryset will be rejected by the server and an error message detailing which columns have been requested from the wrong loa will be returned.The final argument to the Column instance is the name of the raw column to be fetched from the database. If a non-existant column is requested, the queryset will be rejected by the server and an error message detailing which columns are unavailable will be returned.Aggregation/disaggregationThe definition of a queryset must include thetargetlevel of analysis, at which the resulting data will be presented to the user.The definition of each column in a queryset must specify thesourcelevel of analysis, at which the raw data used to define that column is stored in the database.If these loas differ for a given column, the necessary aggregation or disaggregation is performed automatically.If anaggregationis required, the user may choose from the following aggregation functions:sum(default): sums over necessary spatial and time unitsavg: averages over necessary spatial and time unitsmax: takes the maxmimum value over necessary spatial and time unitsmin: takes the minimum value over necessary spatial and time unitscount: counts non-zero values over necessary spatial and time unitsIf no aggregation function is specified but aggregation is required, the default choice (sum) will be automatically (and silently) selected.It is up to users to ensure that they select the correct aggregation functions. If something other than the default is required, an aggregation function can be specified bynew_queryset = (Queryset("simple_conflict", "country_month")

           .with_column(Column("ged_sb", from_loa="country_month", from_column="ged_sb_best_sum_nokgi")
                        .aggregate('avg')
                       )
               )If a non-existent aggregation function is specified, the queryset will be rejected by the server and an error message detailing which columns have incorrect aggregation functions will be returned.TransformsAny queryset column may specify an arbitrary number of transforms to be done to the raw dataafterany necessary aggregation/disaggregation has been done.Transforms are added to columns using thetransformmethodnew_queryset = (Queryset("simple_conflict", "country_month")

           .with_column(Column("ged_sb", from_loa="country_month", from_column="ged_sb_best_sum_nokgi")
                        .aggregate('avg')
                        .transform.ops.ln()
                        .transform.missing.replace_na()
                       )          
               )Each transform method has a general transform type, e.g.missingfollowed by a specific function with brackets for arguments.A list of available transforms can be obtained using the viewser CLI. A notebook giving examples of what each transform does can be found athttps://github.com/prio-data/views3/tree/master/examples.Note that not all transforms are available at all levels of analysis. If a transform is requested at an innapproprite loa, the queryset will be rejected by the server and an error message detailing which columns have requested incompatible transforms and loas will be returned.Publising a querysetBefore a queryset can be fetched, it must be published to a permanent database on the server. This is done using thepublish()method:data = new_queryset.publish()Fetching a querysetA published queryset can be fetched using the.fetch()methoddata = new_queryset.fetch()which can also be chained with thepublish()method:data = new_queryset.publish().fetch()Communication between the viewser client and the server is by a simple polling model. The client sends the queryset to the server again and again with a pause (currently 5 seconds) between each send.Each time, the server responds with one ofa status message informing the user of progress on computing their querysetan error message detailing something that has gone wronga compressed completed datasetThe fetch process proceeds as follows:(i) At the first request, the server enters the queryset into a temporary database of 'in-progress' querysets that it is currently working on. Querysets are removed from this database if they cause an error to be generated, or once the completed dataset has been sent to the client.(ii) Once the queryset has been entered into the temporary database, it is validated to check that, e.g. no non-existent database columns, aggregation functions or transforms have been requested. If this is the case, an error message is sent back to the client and the queryset is deleted from the 'in-progress' database.(iii) If the queryset passes validation, the server compares the requested columns with what is in its cache to see if some or all of some or all of the columns have already been computed. Already computed columns are not computed again, and partially computed columns are 'pruned' of stages that are already in the cache, so that work is not repeated.(iv) If raw data needs to be fetched from the database, a job is dispatched to a database queue, which fetches the missing raw data to the cache. While this is in progress, the server returns status messages to the client detailing whether the database fetch is still waiting in the queue, or its progress if the fetch has started.(v) Once all necessary raw data is in the cache, any transforms that remain to be done are dispatched to a transform queue. During this phase, status messages are returned to the client detailing whether the transforms have been started, and what their progress is.If errors are encountered during the database fetch or transform stages, an error message is returned to the client and the queryset is removed from the 'in-progress' database.(vi) Otherwise, all completed queryset columns are written to the cache, then assembled into a single pandas dataframe, which is compressed into parquet format and sent back to the client. The queryset is removed from the 'in-progress' database.Note that priogrid-level dataframes, even compressed, can be large and can take significant time to download.Common viewser error messagesValidation errorsWhen a queryset is passed to the service, it is examined by a validation function which checks for easily-detected errors. Errors found by the validator will be received immediately by the client:'validation failed with illegal aggregation functions:   [list of bad aggregation functions]' - indicates that one or more non-existent aggregations was requested'validation failed with repeated column names:   [list of repeated column names]' - indicates that one or more column names has been used more than once in the queryset definition'validation failed with non-existent transforms:   [list of bad transforms]' - indicates that one or more non-existent transforms was requested'validation failed with disallowed transform loas: [list of bad transform:loa combinations] - indicates that the transform:loa pairings in the list are illegalRuntime errorsOther kinds of error are only detectable once processing the queryset has started, so these errors may take considerably longer to appear:'db fetch failed - missing columns: [list of bad column names]' - indicates that the listed columns do not exist in the VIEWS database'db fetch failed, to_loa = country_month, columns = ['/base/<bad_loa>.ged_sb_best_sum_nokgi/country_month.sum'], exception = no such loa is available right now!' - indicates that when trying to fetch the column 'ged_sb_best_sum_nokgi', the source loa <bad_loa> does not exist'transform failed,   file (path to transform function on server), line XX, in (transform), (specific error message)' - indicates that a transform operation failed, likely because of non-sensical parameters - the specific error message gives more detailsFundingThe contents of this repository is the outcome of projects that have received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 694640,ViEWS) and Horizon Europe (Grant agreement No. 101055176,ANTICIPATE; and No. 101069312,ViEWS(ERC-2022-POC1)), Riksbankens Jubileumsfond (Grant agreement No. M21-0002,Societies at Risk), Uppsala University, Peace Research Institute Oslo, the United Nations Economic and Social Commission for Western Asia (ViEWS-ESCWA), the United Kingdom Foreign, Commonwealth & Development Office (GSRA –Forecasting Fatalities in Armed Conflict), the Swedish Research Council (DEMSCORE), the Swedish Foundation for Strategic Environmental Research (MISTRA Geopolitics), the Norwegian MFA (Conflict TrendsQZA-18/0227), and the United Nations High Commissioner for Refugees (the Sahel Predictive Analytics project).