Package: blackblox
BlackBloxBlackBlox is a calculator for "black box" systems, ranging from single unit processes to factories with branching chains of processes.Installation and development instructionsInstalling and using BlackBlox(Optional) Install the only non-Python dependency:GraphvizBy following the instructions on theofficial websiteImportant is that thedotexecutable should be in yourPATHDo the normal installation of the library viapip:pip install blackbloxUsage and examplesLook around the configuration options, see what they mean, and whether the defaults seems acceptable to youInblackblox.dataconfig_formatyou find what are the names and meanings of the optionsInblackblox.dataconfig_defaultyou see what are the default valuesLook at the demonstration scenario underscenarios-examples/demo/demo.pyfor inspirationHere you can learn how to change configuration optionsAs well as how to create and balance unit processes, chains, factories, etc.Contributing to BlackBlox development and releasing to PyPIThis project usespoetryas its dependency management, virtualenv management and release (build) toolInstall following the steps described inhttps://python-poetry.org/docs/#installationThe API docs and explanations about all data needed to run scenarios are in thedocsdirectoryWe useSphinxfor building docs. Poetry also ensures that dev dependencies (Sphinx etc.) are installedRun the make command from Sphinx throughpoetry run, so that Sphinx is found from the project's virtualenv
(managed by Poetry)cd docs; poetry run make htmlThe generated pages will be underdocs/_build/htmlBuilding a (new) release and publishing it to PyPI:Register as a contributor (first time only)Make an account onhttps://pypi.org. Ask (optional) for invitation to become project contributor on PyPI.Add API token on the "account settings" page of PyPI (global scope for now)Register the API token to be the one used by Poetry:poetry config pypi-token.pypi "<your_api_token>"Do the actual contribution to the project 🙂Test on local machineRunpoetry installto create a clean environment with needed depedenciesRunpoetry shellto enter a virtual environment from the top level of the project folderTest whatever you want to test within the virtual environmentrunningblackblox --config scenarios-examples/demo/demo.yamlis a good place to startas ispython3 scenarios-examples/demo/demo.pyRunpoetry updateto get possible dependecy updates, and commit the updatedpoetry.lockfile (optional)Increment the package's version number inpyproject.tomlBuild the package (wheel and source):poetry build. The built artifacts will be placed in thedistfolderPublish to PyPI:poetry publishWhat is blackblox.py?Blackblox is an open source python3 library for constructing and solving "black box" process models. Its particular strength is allowing for rapid comparison of different scenarios of parameters. Input data can be stored in Excel workbooks or tab or comma delimited text files. Results are returned as dictionaries for use within python and written to Excel workbooks. Results can also be displayed on the terminal console. Blackblox's basic functionality can be run from the terminal without needing to write any Python script.The basic "block" of blackblox is theUnitProcess, which contains the relationship between the inflows and outflows of a process or subprocess under evaluation. This block can be connected with others to form a linearProcessChain. Units and chains can be further connected in a linear or branching fashion to form aFactorymodel.For each level of model, two main functions are available:balance(): which evaluates the model based on one scenario, given a quantity and a specific flow to balance on.run_scenarios(): which evaluates the model on multiple scenarios, returning comparative results in a single table.Each of these return the results as dictionaries or Pandas DataFrames, which can be used for further process within Python. Results can be written to the console or saved to an Excel file.Additionally, theFactory-level model has support for single-variable sensitivity analysis.Unit ProcessesUnit processes are the smallest "block" in blackblox. Each unit process has a set of inflows and outflows and a set of specified fixed-ratio (linear) relationships between the process flows. Then, given a quantity for one inflow or outflow, the quantities of the remaining inflows and outflows can be calculated.A unit process is defined by two tables, acalculations tablethat specifies relatioships between flows, and avariables tablethat specifies the numerical values for the variable parameters used in the calculations. Extra data about unit process flows can be stored inlookup tables. All unit processes need to be listed in aunit librarytable, which tells blackblox where to find them.Figure: Representation of a unit processCalculations TablesThe calculation tables specifies the relationships between flows in a unit process. Each row in the table:names two flows that have a relationshipdesignates whether each flow is an inflow, outflow, or a flow internal to the processspecifies the type of calculation that would generate the quantity of the second substance if the quantity of the first substance is knownspecifies the variable parameter (if any) used in that calculation that is the same name as a column in the variables table.Table 1 provides an example of a calculation table for a simplified cement kiln.KnownQty(k) andUnknownQty(v) contain the two substances with a known relationship.k_QtyFromandu_QtyTospecify the location of each flow.inflowori: inflow of unit process. Each inflow must be uniquely named and represent a unique calculation.outfloworo: outflow of unit process. Each outflow must be uniquely namedtmport: a process-internal dictionary. This can be used for intermediate calculations and does not appear in inflows or outflows.emissionore: outflow of unit process, which does not have to be unique. E.g. In our example table, both the calcination of limestone and the combustion of fuel will generate CO2. By specifying CO2as theu_QtyTolocation, the CO2from both calculations can be added together. Emission flows cannot be used to balance the unit process.coinfloworc: inflow of unit process, which does not have to be unique. Coinflows cannot be used to balance the unit process.Calculationspecifies the type of calculation performed between the two flows. The calculation types must be those available in the program's calculator library, some of which are described in Table 2.Variable(v) species the column in the Variables Table where to find the value of the variable parameter used in the calculation. It is also possible to define substance names that are "lookup variables" that allow the substance to be defined in the variables table and also have properties defined elsewhere. By default in blackblox, the flow name "fuel" is designated as a lookup variable, corresponding to the "fuelType" column in the Variables Table.By default, flow names that begin or end with "energy", "heat", or "electricity" are assumed to be energy flows. All other flows are assumed to be mass flows.Table: Example of blackblox calculation table for a clinker kilnKnownQtyk_QtyFromUnknownQtyu_QtyToCalculationVariableclinkeroutflowCaOtmpRatioCaO_in_ClinkerCaOtmpCaCO3inflowMolMassRatio1CaCO3inflowclayinflowRemainderCaCO3_in_MealCaCO3tmpCO2emissionMolMassRatioclinkeroutflowfueltmpinflowfuelDemandfuelDemandinflowenergy_from_fueloutflowCombustioncombustEffclinkeroutflowelectricityinflowRatioelecDemandTable: Calculation Types. (k = KnownQty, U = UnknownQty, V = variable)Calculation NameVariable Parameter Type (v)Relationship (k:u)DescriptionRatiofloatk * var = uMultiples the known flow quantity by the variable parameterRemainderfloatk * (1 - var) = uMultiples the known flow quantity by (1 - the variable parameter). Only works if the variable parameter is less than one.Returnvaluefloatk = uReturns the known flow quantity.MolMassRatiofloat or none(Molar Mass of U / Molar Mass of K) * v = UMultiples the known flow quantity by the molar mass ratio of the unknown flow to the known flow. Requires both flows names to be valid chemical formulas. If no value for the variable is provided, it defaults to 1.Combustionfloatk (mass) * (MJ/kg of K) * v = u (energy) OR k (energy) / MJ / (1-v) = u (mass)Using data from a separate "fuels" table, generates the energy provided by combusting a specified amount of fuel, or the fuel required to generate a specified amount of energy, as well as any specified emissions from the combustion. The variable parameter is the combustion efficiency, with the waste heat retuned as a separate flow.e.g. [k OR u] (energy) * (1-v) = wast heat (as emission)Combustion also adds the heat remaining after the combustion efficiency is applied as "waste heat" to the dictionarye.g. [k OR u] (mass) * (kg CO2/kg fuel) = CO2(as emission)Combustion can also look up emission factors from a separate "fuels" table (here shown for CO2, but any emission desired can be added)Additionnonek + k2 = uAdds two known flow quantities together. Requires two columns to be added to the calculations table:2nd Known Substance,2Qty OriginSubtractionnonek - k2 = uAdds two known flow quantities together. Requires two columns to be added to the calculations table:2nd Known Substance,2Qty OriginVariables TablesThevariables tableprovides the values of the variable parameters specified in the calculations table. There is one column for each of the variables named in thecalculation table. The column name must be exactly the same as the specified variable name. Each row is a set of variable parameter values, identified by ascenarioname.Separating the values into their own table allows for the same unit process to be easily evaluated for multiple scenarios of parameter values (e.g. different production efficiencies or fuel types). (note: rows whose scenarios begin with "meta" are ignored by blackblax and can be used for notes. This is also true for columns whose name begins with "meta".)Table: Example of a blackblox variables table for a hypothetical clinker kiln.scenariofuelDemandfuelTypeCaO_in_ClinkerCaCO3_in_MealcombustEffelecDemandmeta-units(mj /t clinker)name(t/t)(t/t)(%)(mj/t clinker)default3coal0.650.810.1EU-old3.6coal0.750.810.2EU-bat_bio3charcoal0.650.810.1EU-typical3.2coal0.670.810.1Lookup tablesBlackblox.py has the ability to get data about flows stored in other spreadsheets. By default, a "fuels" table is a lookup table that is recognized by the combustion calculation to get information about energy content and emission factors.Table: Example Fuels Tablefuel typeLHVCO2__fossilCO2__biometa-sourcemeta-units(GJ/dry tonne)(t/t combusted)(t/t combusted)heavy fuel oil40.43.1270IPCC emission factor databasecoal25.82.47940IPCC emission factor databasenatural gas482.69280IPCC emission factor databasecharcoal29.503.304IPCC emission factor databasecoke28.23.01740IPCC emission factor databaseUnit  LibraryThe unit library is a table that lets blackblox know what unit processes exist and how to identify them. It requires the columns:id: a unique identifier for the unitdisplay name: what the unit is called in the resulting outputproduct: the default product of the unit. This is the flow the unit will be balanced on if no other is specifiedproductType: ("inflow" or "outflow"). Whether the default product is an inflow or outflow of the unit process.Table: Example Unit Libraryiddisplay nameproductproductTypeclinker_kilnkilnclinkeroutflowcement_blenderblendercementoutflowmeal_mixermixermealoutflowelectricityelectricity generationelectricityoutflowgas_scrubberSCR flue gas cleaningflue gasinflowStoring DataBy default, blacblox looks for data in adata/subfolder of the working directory. Data can be in tab or comma delimited text files (.txt, .tsv, .csv) or in the sheets of an Excel workbook (.xls, .xlsx). If Excel workbooks are used, variable tables and calculation tables must be in separate workbooks.The data folder needs to contain the unit process library,unitlibrary.xlsx(or .csv, .tsv, .txt) and any lookup tables, e.g.fuels.csv.Blackblox will look for unit process data in thedata\folder and its immediate subfolder. It identifies files that contain variable or calculation tables by the use of a filename prefix. By default, unit variable data files need to begin withvar_(e.g.var_cement.xls) and calculation data files need to begin withcalc_(e.g.calc_cement.txt`).If using delimited text files to store unit process data, the file name after the prefix should be the same as the unit id listed in the Unit Library. For example,  a process with the idclinker_kilnwould require the filesvar_clinker_kiln.csvandcalc_clinker_kiln.csv.If using excel files to store unit process data, the excel file must begin with the file identifiers (e.g.var_units.xlsxandcalc_units.xlsx) and the sheet names of each file must align with the unit ids from the unit library (e.g.clinker_kilnshould appear as a sheet in eachvar_units.xlsxandcalc_units.xlsxwith the relevant variable table and calculation table, respectively).So a valid file structure for blackblox could look something like this:project/
├─ config.yaml
├─ data/
│  ├─ unitlibrary.csv
│  ├─ fuels.csv
│  ├─ units/
│  │  ├─ var_units.xlsx
│  │  ├─ calc_units.xlsx
│  │  ├─ var_unitA.csv
│  │  ├─ calc_unitA.csvCreating and balancing a unit processesBalancing a unit process calculates the quantity of all inflows and outflows of that unit process for a given set of variable parameters. To balance a unit process, the following information is needed:the quantity of one inflow or outflow.  (if not specified, defaults to 1.0)the name of that inflow or outflow substance (if not specified, defaults to the product listed in the unit library)whether the substance is an inflow or outflow (if not specified, defaults to the productType listed in the unit library)the name of the scenario to use from the variables table (if not specified, defaults todefault.)After calculating all flows based on the user input, blackblox then checks for whether the mass and energy flows are balanced, and either raises an exception or, by default, creates an 'UKNOWN' balancer flow where needed. When processing combustion-type calculations, blackblox writes an "energy in combustedfuelType" flow to the inflows dictionary, to balance the energy of combustion. A similar balancer flow is provided for the combustion emissions.Balancing a unit process in pythonUnit Processes need to be created and then can be balanced. In python, the unit process can be created via:blackblox.unitprocess.UnitProcess(unit_id)So, to create and balance the clinker kiln of our example, you could create it usingkiln = blackblox.unitprocess.UnitProcess(clinker_kiln)and then balance it withkiln.balance(scenario='EU-typical', qty=100)The results are returned as a dictionary and can be displayed in the console usingwrite_to_console=Trueor written to an excel sheet usingwrite_to_xls=True.To compare multiple scenarios, the functionrun_scenarioscan be used. By default, the results are written to an Excel file.kiln.run_scenarios(scenario_list=['EU-typical', 'EU-old', 'EU-bat_bio'])Process ChainA process chain is a linear set of connected unit processes, where an outflow of a preceding unit process is an inflow of the following unit process.Defining a chainA process chain is defined by a table with a list of unit processes with an inflow and outflow to each, where the outflow of a unit process must be the inflow into the next unit process, as shown in the below table.Table:  Example chain tableInflowProcess_IDOutflowCaCO3mixermealmealkilnclinkerclinkerblendercementWhen the process chain is first used, an initialisation process creates each of the unit processes, if they do not already exist, and verifies that the inflows and outflows specified in the chain table exist for the corresponding unit processes.A process chain can be defined by specifying the location of the chain table either in an Excel workbook or delimited text file usingchainName = processchain.ProcessChain(chain_data='path/to/chaindata.xlsx, 
                                       xls_sheet='sheet name)(xls_sheetis only required if using an Excel file with multiple sheets)Balancing a chainBalancing a chain calculates the quantity of all inflows and outflows of each unit process in the chain, either from first inflow to last outflow or from last outflow to first inflow. To balance a chain, the following arguments  must be provided:qty: the quantity of one inflow or outflow somewhere in the chainproduct: the name of that inflow or outflow substance. (Attempts to default to product of last unit process, if an outflow. If not, will attempt to default to the product of the first unit process, if an inflow. )i_o: whether the product is an inflow or outflow (defaults to the default product location)scenariothe name of the configuration scenario to use from the variables table. (defaults to "default")the name of the unit process in the chain, if the specified flow is not an inflow into the first process or an outflow of the last process.All arguments besides the quantity can be optional, if a default product can be determined.Balancing a chain returns a dictionary with both the calculated inflows and outflows for each unit process, as well as the the overall inflows and outflows. The results can also be written to the console or an excel file.Generating a chain diagramAfter a chain has been defined, a process flow diagram of the chain can be generated automatically usingprocessName.diagram()Ff Graphviz has been correctly installed to your computer's path. If not, this will fail with a hopefully useful error message.FactoryA factory is a collection of one or more connected process chains, where the inflow of outflow of any unit process within any chain can be the inflow or outflow of any other chain. A factory has a single main chain, and zero or more auxiliary chains. By specifying an input or output quantity to the main chain, it is possible to calculate the inflows and outflows of all processes within the chain.Defining a factoryA factory is defined by two tables: one that lists the chains (and their file locations) and another that lists the connections between the chains. Factory data can be in separate tab delimited text files or in a single excel workbook.Factory chains tableThis table specifies the location of the process chain data used in the factory, as well as the  primary product and whether that product is an inflow or outflow of the chain.The first chain of the chain list is assumed to be the main product chain, but otherwise it doesn't matter.
The chains can either be in the same file, if an excel workbook, or in a separate file.Table: Example of a Factory Chains TableChainNameChainProductProduct_IOChainFileChainSheetcementcementoutflowhereCement ChainCO2captureCO2inflowhereCO2 CapturepowerelectricityoutflowherePower ChainFactory connections tableThe factory connections table specifies how the chains connect, including the origin process chain and unit process, the destination chain, the connecting product, and whether the product is an inflow or outflow of the origin and destination.
The connections table is order dependent, and the first chain should always be the main chain.Table: Example of a Factory Connections TableOriginChainOriginProcessProductProduct_IO_of_OriginProduct_IO_of_DestinationDestinationChaincementkilnCO2outflowinflowCO2capturecementallelectricityinflowoutflowpowerCO2captureallelectricityinflowoutflowpowerIf the destination is not the start or end of a chain, an optionaldestination processcolumn can be used. This column is also used when specifying recycling connections. Currently two types of recycling are usable:1-to-1 replacement: allows for a recycle flow to replace a flow in another unit process, given that is a 1-to-1 correspondent and does not affect other flows in unit process.energy replacing fuel: allows for recycled energy to replace energy generated from a fuel in a unit process that has a combustion calculation. The emission outflows (e.g. CO2) and inflow of oxygen are also recalculated.If there is a greater quantity of the recycled flow than is used in the destination process, the remainder is treated as a normal output.  A single flow can be recycled into multiple units, by specifying each connection as a new line in the table.  If there is insufficient recycling flow to fully replace the existing flow, it is only partially replaced.Balancing a factoryBy default, the factory is balanced on a quantity of the product of the main ProductChain, and uses a single scenario of variables for the whole factory.factory.balance()takes the same parameters asproductchain.balance(). Additionally, the factory can be balanced in other products from the main chain by passingproduct,product_unit,product_ioparameters. A factory cannot be balanced on auxiliary chains.Balancing a factory returns a nested dictionary with the inflows and outflows of the factory, every chain, and every unit process. By default, the results are saved to an Excel workbook with sheets for:Total factory inflows and outflowsA matrix for all inflow from every unit processA matrix for all outflows from every unit processA list of all internal flows/connections, both within chains andInflows for each chain unit process and chain totals (per chain)Outflows for each chain unit process and chain totals (per chain)By default, the data is divided into mass and energy flows, if energy flow name signifiers are provided, with totals for both mass and energy flows.Generating a factory diagramAfter a factory has been defined, a process flow diagram of the factory can be generated. However, due to the limitations of the diagram rendering software, they are unlikely to be pretty. Graphviz must be properly installed for this to work.Sensitivity AnalysesA single-variable sensitivity analysis can also be run on a Factory model.  Thefactory.run_sensitivity()function takes all the same parameters asfactory.balance()as well as:unit_name: (str) the identifier of theUnitProcesscontaining the variable of interestchain_name: (str) theProcessChaincontaining the unit of interestvariable: (str) the variable of interest, which should correspond to a column in the unit'sVariable Tablevariable_options: a list of the variables options to be calculated.The sensitivity analysis returns an Excel workbook with results for each variable option in columns.Where to Find blackblox.pyBlackblox.py is an free and open source library released under the GNU General Public License v3 (GPLv3). Blackblox is under active development. Currently available features and documentation may be different than what is provided in this Appendix.Full documentation for blackblox can be found at the project homepage:https://concoctions.org/blackblox/Source code and data are available on GitHub:https://github.com/concoctions/BlackBloxBlackblox can be installed usingpip install blackblox. Information about the package can be found at PyPi:https://pypi.org/project/blackblox/