Package: bw-processing
bw-processingLibrary for storing numeric data for use in matrix-based calculations. Designed for use with theBrightway life cycle assessment framework.Table of ContentsBackgroundConceptsInstallUsageContributingMaintainersLicenseBackgroundTheBrightway LCA frameworkhas stored data used in constructing matrices in binary form as numpy arrays for years. This package is an evolution of that approach, and adds the following features:Consistent names for row and column fields. Previously, these changed for each matrix, to reflect the role each row or column played in the model. Now they are always the same for all arrays ("row"and"col"), making the code simpler and easier to use.Provision of metadata. Numpy binary files are only data -bw_processingalso produces a metadata file following thedata package standard. Things like data license, version, and unique id are now explicit and always included.Support for vector and array data. Vector (i.e. only one possible value per input) and array (i.e. many possible values, also called presamples) data are now both natively supported in data packages.Portability. Processed arrays can include metadata that allows for reindexing on other machines, so that processed arrays can be distributed and reused. Before, this was not possible, as integer IDs were randomly assigned on each computer, and would be different from machine to machine or even across Brightway projects.Dynamic data sources. Instead of requiring that data for matrix construction be present and savedd on disk, it can now be generated dynamically, either through code running locally or on another computer system. This is a big step towards embeddding life cycle assessment in a web of environmental models.UsePyFilesystem2for file IO. The use of this library allows for data packages to be stored on your local computer, or onmany logical or virtual file systems.Simpler handling of numeric values whose sign should be flipped. Sometimes it is more convenient to specify positive numbers in dataset definitions, even though such numbers should be negative when inserted into the resulting matrices. For example, in the technosphere matrix in life cycle assessment, products produced are positive and products consumed are negative, though both values are given as positive in datasets. Brightway used to use a type mapping dictionary to indicate which values in a matrix should have their sign flipped after insertion. Such mapping dictionaries are brittle and inelegant.bw_processinguses an optional boolean vector, calledflip, to indicate if any values should be flipped.Separation of uncertainty distribution parameters from other data. Fitting data to aprobability density function(PDF), or an estimate of such a PDF, is only one approach to quantitative uncertainty analysis. We would like to support other approaches, includingdirect sampling from real data. Therefore, uncertainty distribution parameters are stored separately,  only loaded if needed, and are only one way to express quantitative uncertainty.ConceptsData packagesData objects can be vectors or arrays. Vectors will always produce the same matrix, while arrays have multiple possible values for each element of the matrix. Arrays are a generalization of thepresamples library.Data needed for matrix constructionVectors versus arraysPersistent versus dynamicPersistent data is fixed, and can be completely loaded into memory and used directly or written to disk. Dynamic data is only resolved as the data is used, during matrix construction and iteration. Dynamic data is provided byinterfaces- Python code that either generates the data, or wraps data coming from other software. There are many possible use cases for data interfaces, including:Data that is provided by an external source, such as a web serviceData that comes from an infinite python generatorData from another programming languageData that needs processing steps before it can be directly inserted into a matrixOnly the actual numerical values entered into the matrix is dynamic - the matrix index values (and optional flip vector) are still static, and need to be provided as Numpy arrays when adding dynamic resources.Interfaces must implement a simple API. Dynamic vectors must support the python generator API, i.e. implement__next__().Dynamic arrays must pretend to be Numpy arrays, in that they need to implement.shapeand.__getitem__(args)..shapemust return a tuple of two integers. The first should be the number of elements returned, though this is not used. The second should be the number of columns available - an integer. This second value can also beNone, if the interface is infinite..__getitem__(args)must return a one-dimensional Numpy array corresponding to the columnargs[1]. This method is called when one uses code likesome_array[: 20]. In our case, we will always take all rows (the:), so the first value can be ignored.Here are some example interfaces (also given inbw_processing/examples/interfaces.py):importnumpyasnpclassExampleVectorInterface:def__init__(self):self.rng=np.random.default_rng()self.size=self.rng.integers(2,10)def__next__(self):returnself.rng.random(self.size)classExampleArrayInterface:def__init__(self):rng=np.random.default_rng()self.data=rng.random((rng.integers(2,10),rng.integers(2,10)))@propertydefshape(self):returnself.data.shapedef__getitem__(self,args):ifargs[1]>=self.shape[1]:raiseIndexErrorreturnself.data[:,args[1]]Interface dehydrating and rehydratingSerialized datapackages cannot contain executable code, both because of our chosen data formats, and for security reasons. Therefore, when loading a datapackage with an interface, that interface object needs to be reconstituted as Python code - we call this cycle dehydration and rehydration. Dehydration happens automatically when a datapackage is finalized withfinalize_serialization(), but rehydration needs to be done manually usingrehydrate_interface(). For example:fromfs.zipfsimportZipFSfrombw_processingimportload_datapackagemy_dp=load_datapackage(ZipFS("some-path.zip"))my_dp.rehydrate_interface("some-resource-name",ExampleVectorInterface())You can list the dehydrated interfaces present with.dehydrated_interfaces().You can store useful information for the interface object initialization under the resource keyconfig. This can be used in instantiating an interface if you passinitialize_with_config:fromfs.zipfsimportZipFSfrombw_processingimportload_datapackageimportrequestsimportnumpyasnpclassMyInterface:def__init__(self,url):self.url=urldef__next__(self):returnnp.array(requests.get(self.url).json())my_dp=load_datapackage(ZipFS("some-path.zip"))data_obj,resource_metadata=my_dp.get_resource("some-interface")print(resource_metadata['config'])>>>{"url":"example.com"}my_dp.rehydrate_interface("some-interface",MyInterface,initialize_with_config=True)# interface is substituted, need to retrieve it againdata_obj,resource_metadata=my_dp.get_resource("some-interface")print(data_obj.url)>>>"example.com"PoliciesData package policies define how the data should be used. Policies apply to the entire data package; you may wish to adjust what is stored in which data packages to get the effect you desire.There are two policies that apply to all data resources:sum_intra_duplicates(defaultTrue): What to do if more than one data point for a given matrix element is given in eachvector or array resource. If true, sum these values; otherwise, the last value provided is used.sum_inter_duplicates(default:False): What to do if data from a given resource overlaps data already present in the matrix. If true, add the given value to the existing value; otherwise, the existing values will be overwritten.There are three policies that apply only to array data resources, where a different column from the array is used in matrix construction each time the array is iterated over:combinatorial(defaultFalse): If more than one array resource is available, this policy controls whether all possible combinations of columns are guaranteed to occur. IfcombinatorialisTrue, we useitertools.combinationsto generate column indices for the respective arrays; ifFalse, column indices are either completely random (with replacement) or sequential.Note that you will getStopIterationif you exhaust all combinations whencombinatorialisTrue.Note thatcombinatorialcannot beTrueif infinite array interfaces are present.sequential(defaultFalse): Array resources have multiple columns, each of which represents a valid system state. Default behaviour is to choose from these columns at random (including replacement), using a RNG and the data packageseedvalue. IfsequentialisTrue, columns in each array will be chosen in order starting from column zero, and will rewind to zero if the end of the array is reached.Note that ifcombinatorialisTrue,sequentialis ignored; instead, the column indices are generated byitertools.combinations.Please make sure you understand howcombinatorialandsequentialinteract! There are three possibilities:combinatorialandsequentialare bothFalse. Columns are returned at random.combinatorialisFalse,sequentialisTrue. Columns are returned in increasing numerical order without any interaction between the arrays.combinatorialisTrue,sequentialis ignored: Columns are returned in increasing order, such that all combinations of the different array resources are provided.StopIterationis raised if you try to consume additional column indices.InstallInstall using pip or conda (channelcmutel). Depends onnumpyandpandas(for reading and writing CSVs).Has no explicit or implicit dependence on any other part of Brightway.UsageThe main interface for using this library is theDatapackageclass. However, instead of creating an instance of this class directly, you should use the utility functionscreate_datapackageandload_datapackage.A datapackage is a set of file objects (either in-memory or on disk) that includes a metadata file object, and one or more data resource files objects. The metadata file object includes both generic metadata (i.e. when it was created, the data license) and metadata specific to each data resource (how it can be used in calculations, its relationship to other data resources). Datapackages follow thedata package standard.Creating datapackagesDatapackages are created usingcreate_datapackage, which takes the following arguments:dirpath:strorpathlib.Pathobject. Where the datapackage should be saved.Nonefor in-memory datapackages.name:str: The name of the overall datapackage. Make it meaningful to you.id_:str, optional. A unique id for this package. Automatically generated if not given.metadata:dict, optional. Any additional metadata, such as license and author.overwrite:bool, defaultFalse. Overwrite an existing resource with the samedirpathandname.compress:bool, defaultFalse. Save to a zipfile, if saving to disk.Calling this function return an instance ofDatapackage. You still need to add data.ContributingYour contribution is welcome! Please follow thepull request workflow, even for minor changes.When contributing to this repository with a major change, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository.Please note we have acode of conduct, please follow it in all your interactions with the project.Documentation and coding standardsBlack formattingSemantic versioningMaintainersChris MutelLicenseBSD-3-Clause. Copyright 2020 Chris Mutel.