Package: nexinfosys
nis-backendAMuSIASEMinstanceNIS (Nexus Information System) is a software system developed initially under the framework of MAGIC project (H2020 grant #689669).NIS enables accounting the biophysical flows in complex bioeconomic systems according to MuSIASEM concepts and methodology, as a way to assess the sustainability of current -or future- scenarios in socio-ecologic systems.NIS is made of four components:nis-backend,nis-frontend,nis-clientandnis-eda.nis-backendis the backend component of NIS, which is deployed embedding another NIS component,nis-frontend(an Angular web application) and it is (programmatically) actionable through a third,nis-client, using Python or R.nis-edais an exploratory data analysis tool written in R and Shiny which connects to anis-backend.nis-backend"runs" MuSIASEM models byinterpreting(lexically, syntactically and semantically) andresolving(quantitatively) a representation of a MuSIASEM grammar (an open model), using atabular syntax("dataframe" compatible), and producing a set of outputsexportablefor further use by external, complementary tools.It is an open system as it can be used to integrate MuSIASEM as a formalism in analytic/scientific workflows thanks to the RESTful API (and the easier to usenis-client, for Python and R scripts).Disclaimer: this README is still under elaboration, details may be missing or innacurate.Table of ContentsDocumentationGetting startedFeaturesInstalling and executingnis-backendPip packageDocker imageSource codeModels with Commandsnis-frontendquick introAccessingnis-backendin Python and R scripts withnis-clientConfigurationContributingPeopleContactLicenseAcknowledgementsKeywordsDocumentationFull documentation:How-to  (Annex 4 - "NIS frontend how-to manual").Technical reference (Annex 2 - "Report on NIsys toolset development").Format of input files (Annex 3 - "Format for specification of NIS case studies").is available in the annexes of the deliverable of the project regarding "nis-backend", in the official web page, athttp://www.magic-nexus.eu/documents/deliverable-33-report-datasets-and-software-tools.NOTE: the "Format of input files" document is updated outside of the main document.Getting startedFeaturesMuSIASEM adaptation. MuSIASEM concepts have been adapted to an object model using a DSL (Domain Specific Language) with tabular syntax (spreadsheet, dataframe).Formalization, standardization. The former point has as consequence a formal and standard definition of MuSIASEM grammars/models, for repeatability, reproducibility, replicability, findability of case studies and of MuSIASEM itself, and also for automatic processing of specifications.Uses. Apt for both learning and teaching; for research, economic analysis, decision making, policy making, diagnosis and prognosis, etc.Integrability. Easy to integrate into analytical workflows (GIS, statistical, engineering, ...).Inputs - External. Integrated use of statistical data sources: Eurostat, OECD, FAO, FADN.Inputs - Internal. Integration of data from custom external spreadsheets.Reusability of model parts. Similar to libraries/packages in programming languages.Integrated solver. Capable of exploiting available quantifications and relationships, in multiple:Systems, e.g. interconnected countries.Scales, e.g. food and macronutrients.Scenarios, which are named sets of values for parameters, e.g. "optimistic", "expected", "pessimistic".Sources, so several observations can be made about the same fact.Times, in years or in months.Outputs. Exportable as datasets (statistical cubes), graphs (networks), matrices, models (transformations of the input model), geolayers, scripts (Python or R).Deployment. It can be deployed in multiple ways: personal computer (pip, source, Docker), or server (source or Docker).Configurable. Deployment for server can be configured to use different database servers and REDIS for session management.Open. All functionality can be accessed through a RESTful service. Behind the scenes, because HTTP protocol is stateless, a complex serialization/deserialization function combined with a key/value store and a browser session enable saving the state in memory after a service call ends, just to recover it when another invocation is done.Two expertise levels. Two components wrap the RESTful interface,nis-frontendandnis-client. These components match two expertise levels:nis-frontenddoes not require programming knowledge, whilenis-clientcan be used in Python/R scripts.Installing and executing "nis-backend"nis-backendis a Python package. There are several options to have it installed.pip packageThe pip version is obviously for execution as package, and does not include the frontend, the other two deployment options below include the frontend.Set a Python environmentInstall the package withpip install nexinfosysUse classNIS, with methods similar toNISClientclass innexinfosys-clientpackage.Docker imageFor a lab setup, this option allows to configure and run nis-backend inside a Docker environment (Docker must be installed in the target computer).The instructions are at "Docker Hub" site, please follow the link:https://hub.docker.com/r/magicnexush2020/magic-nis-backendSource codeNOTE: Python3 and git required.Clone this repository and execute "service_main.py":git clone https://github.com/MAGIC-nexus/nis-backend
cd nis-backend
git checkout develop
pip install -r requirements.txt
PYTHONPATH=. python3 nexinfosys/restful_service/service_main.pyAlternately to the last line, for a more robust server (gunicornassumed to be installed):gunicorn--workers=3--log-level=2000--bind0.0.0.0:5000nexinfosys.restful_service.service_main:appChange the port to the one desiredModels with CommandsTo specify MuSIASEM grammars (models), NIS proposes a syntactically simple "block oriented DSL" (DSL - Domain Specific Language), structured in a spreadsheet, where each worksheet is considered a batch of Commands of the same kind, each row is an instance of the command and each column is a field for the command.In order to take advantage ofnis-backend, analysts write a workbook considering that commands (worksheets) are read and executed from left to right, and inside a worksheet from the first row to the last.The reference of a command appears when a command from the "Commands" tab is selected innis-frontend. A document with a complete reference is available at ¿ZENODO?. An HTML called "Commands reference" can be obtained from "Exportable datasets" tab, containing an up-to-date reference.The next section introducesnis-frontendwhich is a tool embedded innis-backendwhose goal is to help in writing MuSIASEM grammars.Quick intro tonis-frontendnis-backendembeds the compiled version of a full Angular application callednis-frontendwhich is the indicated interface tonis-backendfor most users.Oncenis-backendis executing in your computer, open a browser a go to the address:http://localhost:5000/nis_clientA screen similar to the following should open:The screen has a top bar (blue colored) where a menu, the Play button and a field allowing the specification of a URL are present.The rest of the screen is divided in two zones, at left and right.The right zone is where the process of representation of a MuSIASEM grammar (model) can be edited (textually or graphically), issues are showed for correction, and outputs can be explored or exported for further elaboration.The left zone contains helpers allowing the insertion of empty Commands (worksheets) and the selection of external Datasets for the elaboration of internal ones.The URL field allows editing a workbook using Google Sheets (the application must be authorized first).nis-backendreads directly from the specified worksheet, so it needs access enabled to the file.For extensive documentation see thedeliverable. Online help for commands appears below the list of commands:and help for each field in a command appears when clicking on the name of a field in a worksheet:When information is typed for a field in the embedded worksheet, syntactic validity is checked.IMPORTANT:nis-frontendis closed source due to a restriction in the license of a commercial component.Accessingnis-backendin Python and R scripts with nis-clientnis-backendis, from a software engineering point-of-view, a RESTful service. All its functionality is accessed through this interface.nis-frontendmakes use of it behind the scenes, it is not necessary for users to know the technical details in this regards. The other alternative designed to access this service is a Python component called "nexinfosys-client". It can be installed using "pip":pip install nexinfosys-clientWith this package it possible to integrate the MuSIASEM related capabilities ofnis-backendin a processing workflow. Seenis-clientREADME for basic usage example.Thanks to R capabilities to use Python packages, the component can be used from R, see also the former README.ConfigurationIf no configuration file is provided (environment variable MAGIC_NIS_SERVICE_CONFIG_FILE not specified), a default configuration file is generated which can be later modified with a text editor.To specify a custom configuration, a text file with the typical syntax of aVariableand itsValueper line must be created:VAR1="VALUE"VAR2="VALUE"Variable nameValueExampleDB_CONNECTION_STRINGMetadata database, SQLAlchemy compliant connection string"sqlite:///nis_metadata.db"DATA_CONNECTION_STRINGDataset cache database, SQLAlchemy compliant connection string"sqlite:///nis_cached_data.db"CASE_STUDIES_DIRDirectory where case studies would be stored"/srv/nis_data/cs/"FAO_DATASETS_DIRDirectory where FAO datasets are downloaded and cached"/srv/faostat/"FADN_FILES_LOCATIONDirectory where FADN datasets are downloaded and cached"/srv/fadn"CACHE_FILE_LOCATIONDirectory where SDMX datasets are downloaded and cached"/srv/sdmx_datasets_cache"REDIS_HOST_FILESYSTEM_DIRIf REDIS_HOST='filesystem:local_session', directory where sessions are stored"/srv/sessions"SSP_FILES_DIRNot used""NIS_FILES_LISTA comma-separated list of URLs to CSV files where NIS case studies or parts of them are enumerated. Each CSV file starts with a header, with four columns: name, url, description and example (True if it is an example)""REDIS_HOST"localhost" expects a REDIS server available at localhost:6379; "redis-local" creates a local REDIS instance; "filesystem:local_session" uses filesystem to store sessions (a good option for execution in PC/laptop)""TESTING"True""True"SELF_SCHEMAName of the host where Backend RESTful service responds, preceded by the protocol (http or https)"https://one.nis.magic-nexus.eu/"FS_TYPE"Webdav""Webdav"FS_SERVERHost name of the WebDAV server"nextcloud.data.magic-nexus.eu"FS_USERUser name used. Files and folders must be readable and writable by this user""FS_PASSWORDPassword for the previous user""GAPI_CREDENTIALS_FILEPath to a file obtained from Google API management web, to directly access a NIS workbook file in Google Sheets"/srv/credentials.json"GAPI_TOKEN_FILEPath to a file used to stored authorization token"/srv/tocken.pickle"ContributingContributions are welcome. Follows a list of possible ways in which you could do it:Testing as end user. Install "nis-backend" (this repository) and execute it, launching the user interface to specify and submit case studies, maybe with the help of an external spreadsheet software. Use Github issues when you find incorrect behavior. Please followMastering Issuesguidelines.Testing as scripts developer. Install "nis-backend" and use it in your scripts using either NIS class or the subprojectnis-clientconnecting it to a running "nis-backend".Prepare reusable MuSIASEM library files, containing model parts, like structures of high level functions in different domains (water, energy, food), interface type hierarchies, technical data about structurals, mappings from specific official statistical datasets to MuSIASEM interface types, calculation of scalar indicators, benchmarks, etc. All of them may be compiled in separate files, as long as the composition of them is consistent.If you are a developer willing to contribute to "nis-backend", please mail one of the contact persons (see Contact below) to discuss some of the features and improvements which could be interesting to have implemented. Once you are ready to contribute, please do it through Github Pull Requests, following thestandard PR workflow.The standardcontributor covenantcode of conduct is earnestly applied in interactions with the community.PeopleAn enumeration of people who have contributed in different manners to the elaboration of NIS during MAGIC project lifetime:Rafael Nebot. ITC-DCCT (Instituto Tecnológico de Canarias - Departamento de Computación).Marco Galluzzi.Michele Staiano. UniNa (Università degli Studi di Napoli Federico II). NIS supervisor, mentor, catalyzer.Paula Moreno. ITC-DCCT.Mario Giampietro, Ansel Renner, Violeta Cabello, Cristina Madrid, Maddalena Rippa, Juan Cadillo, Raúl Velasco, Louisa di Felice, Sandra Bukkens. ICTA-UAB (Institut de Ciència i Tecnologia Ambientals - Universitat Autònoma de Barcelona). MuSIASEM creators and mentors, analysts.Ignacio López.Alberto Sosa, Francisco Socorro, María Artiles, Carlos Caraballo, Ivet Cabrera. Internship students at ITC-DCCT.ContactPlease send any question regarding this repository tornebot@itccanarias.org.LicenseThis project is licensed under the BSD-3 License - see theLICENSEfile for detailsAcknowledgementsThe development of this software was supported by the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 689669 (MAGIC). This work reflects the authors' view only; the funding agencies are not responsible for any use that may be made of the information it contains.KeywordsSustainability - Bioeconomy - Socio-Ecological Systems - Complex Adaptive Systems - Water-Energy-Food Nexus