Package: pinns
Pinns is a python library which creates neural networks that can solve differential equations.DescriptionPinns implements the emerging and promising technology of physics-informed neural networks. It provides an interface for the easy creation of neural networks, specifically designed and trained for solving differential equations. It is build on Tensorflow and Keras libraries.By utilizing the tools it provides, a programmer or a mathematician can build a model easily with few lines of code, and train it in a few minutes. When trained, it can solve the equation for any input instantly. The library has been designed to support many categories of differential equations, as well as a big variety of initial and boundary condition types.Future versions aim to extend the variety of equations categories that can be solved, while making the training process faster.Table of contentsWhat are Physics Informed Neural Networks?Install the pinns libraryDefine the domainMethod get()Create the loss function methodFirst derivativesSecond derivativesHigher derivativesForms and functions of x and yExamples of complete pde methodsDefine the initial conditionsTransformation of initial conditionsCreation of initial conditionsBuild and train your modelBuild the neural networkTrain the modelComplete implementation examplePrint the resultsLicenseAuthorWhat are Physics Informed Neural Networks?In short, Physics Informed Neural Networks (PINNs) are neural networks (NN) that can solve differential equations (DE). They belong in the field of Physics-Informed Machine Learning, which aims to simulate physical phenomena with deep learning algorithms that incorporate empirical data and mathematical models.A PINNs operation is summarized as follows: Data x are fed into the input layer, and an approximation of the solution y comes out of the output layer. Then, using back propagation it calculates all the derivatives present in the DE, using the x and y data. This process is called automatic differentiation. Having the x,y, and derivative data, it formulates the standard from of the DE (right side of the equation equals zero), and sets it as a loss function. Since the NN naturally tries to get the loss function as close to zero as possible, by training the NN it simultaneously finds an approximation of the DE solution.Install the pinns libraryThe library is available at the PyPi repository, so you can easily install it by typing:!pip install pinnsDefine the domainThe domain and all things related to point sets are handled by thegeometrymodule. The DE domain is created by theDomainclass. The constructor has 3 parameters:minval(int/list):Lower boundary coordinatesmaxval(int/list):Higher boundary coordinatesnum_domain(int):Number of domain pointsThe boundary coordinates for an ordinary DE are numbers, whereas for a partial DE are lists of numbers. The number of domain points can vary from a few hundred to a few thousands, depending on the difficulty and complexity of the DE, so feel free to experiment with the number. Keep in mind however that the more points the domain has, the longer will the training take.Example 1:Simple interval $x \in [-1,1]$, with 100 points:x = pinns.Domain(-1,1,100)For multidimensional domains, the lower and higher boundaries for each input variable are in lists.Example 2:Domain $x_1 \in [-1,1], x_2 \in [0,2]$, with 100 points:x = pinns.Domain([-1,0],[1,2],100)Note 1:The domain created by the above command is not to be confused with the domain $x_1\in{[-1,0]}, x_2\in{[1,2]}$. The separation of input parameters is based on the boundaries of the variable intervals, and not by the intervals themselves.Note 2:The library currently supports only domains whose individual variables are intervals. That means that more complex geometrical domains like circles, spheres, polygons and triangles are not yet supported.Method get()The get() method is used on Domain objects, in order to acquire a specific vector from the domain. For example, if we want to access separately one of the input variables from the above example, we type:x0 = x.get(0)
x1 = x.get(1)Note 1:Watch out for the numbering used for the DE variables. In many mathematical statements the numbering starts from 1, but the first element in the Domain is returned with get(0). In the example above, the original domain variables were $x_1$ and $x_2$, but to avoid confusion when handled in the code, they were renamed tox0andx1respectfully.Note 2:Keep in mind that while the original x variable is a Domain object, variablesx0andx1are tensorflow.Variable tensors, so properties of Domain do not apply to them.Create the loss function methodThe loss function method is created by the user just like any other python method. For the rest of the doc we'll be conventionally calling itpde(). It needs to have 2 parameters,xandy, and return the standard form of the DE. The DE contains different forms ofxandy, and also their derivatives, which need to be calculated. Derivatives are handled by thedifferentiationmodule.First derivativesThe first derivatives are calculated with thejacobian()method of classGradients, orGradfor short. It takes 4 parameters:y:Vector of dependent variablesx:Vector of independent variablesyi(int):Index of dependent variable for differentiationxi(int):Index of independent variable for differentiationTheyandxarguments are the same as the arguments of thepdemethod.Indexyispecifies the $y$ variable being differentiated. For DEs with one output variable, this index is always 0, which is also the default value. The argument is needed when we have a system of DEs with multiple $y$ outputs, like $y_1, y_2 ...$ etc.Indexxispecifies the $x$ variable with respect to which the derivative is performed. For ordinary DEs with one input, the index is always 0, which is also the default value. The argument is needed when we have a partial DE with multiple $x$ inputs, like $x_1,x_2...$ etc.Example 1:Derivative $y'(x)$ from an ordinary DEdyx = pinns.Grad.jacobian(y,x)Example 2:Derivative $dy_0 \over dx_2$ from a DE systemdy1x2 = pinns.Grad.jacobian(y,x,yi=0,xi=2)Note:Again, mind that despite the numbering used in the mathematical formula, the first variables in python start from 0.Second derivativesThe second derivatives are calculated with thehessian()method of classGradients. It takes 5 parameters:y:Vector of dependent variablesx:Vector of independent variablesyi(int):Index of dependent variable for differentiationxi(int):Index of independent variable for the first derivativexj(int):Index of independent variable for the second derivativeTheyandxarguments are again the same as the arguments of thepdemethod, and theyiindex has the same use as in thejacobianmethod.
Indicesxiandxjspecify the $x$ variables with respect to which the first and second derivatives are performed. For ordinary DEs with one input, both indices are always 0, which is also the default value.Example 1:Derivative $y''(x)$ from an ordinary DEdyxx = pinns.Grad.hessian(y,x)Example 2:Derivative $d^2y_1 \over dx_1 dx_2$ from a DE systemdy1x1x2 = pinns.Grad.jacobian(y,x,yi=1,xi=1,xj=2)Higher derivativesIn order to compute derivatives of higher order than 2, you use a combination ofjacobianandhessianmethods. The result of one method must be fed as ayargument to the other.Example 1:$\ y^{(3)}(x)$dyx = pinns.Grad.jacobian(y,x)
dyxxx = pinns.Grad.hessian(dydx,x)Example 2:$\ d^3y_2 \over dx_1^2dx_2$dy2x1x1 = pinns.Grad.hessian(y,x,yi=2,xi=1,xj=1)
dy2x1x1x2 = pinns.Grad.jacobian(dy2dx1x1,x,yi=0,xi=2)Note:An already calculated derivative likedy2x1x1consists of only one variable y (in this case $y_2$), so when its used as a parameter, theyiindex must be 0.Forms and functions of x and yInside the DE formula can exist different variables of x and y, which we can access with theget()function. There can also be mathematical functions of them like trigonometric equations. For these equations it's better to use methods provided by the Tensorflow library to avoid any unwanted errors, since the tensors representing the variables are Tensorflow based. For constants like π, Numpy can also be used.Example:$\ q(x,y) = y_1 - e^{-x_0}sin(x_1)$import tensorflow as tf
x0 = x.get(0)
x1 = x.get(1)
y1 = y.get(1)
qx = y1 - tf.math.exp(-x0)*tf.math.sin(x1)Examples of complete pde methodsExample 1:Let be DE: $y^{(3)} = 2$Standard form: $y^{(3)}-2=0$def pde(x, y):  
    dyxx = pinns.Grad.hessian(y, x)  
    dyxxx = pinns.Grad.jacobian(dy_xx, x)  
    return dy_xxx - 2Example 2:Let be DE system: ${{dy_1 \over dt} = y_2}, {{dy_2 \over dt} = -y_1}$Standard forms of DEs: ${{dy_1 \over dt} - y_2} = 0, {{dy_2 \over dt}  +y_1} = 0$def pde(x,y):
	dy1t = pinns.Grad.jacobian(y,x,yi=0)
	dy2t = pinns.Grad.jacobian(y,x,yi=1)
	y1 = y.get(0)
	y2 = t.get(1)
	return [dy1t - y2, dy2t + y1]Note:When we have a system of DEs, the return statement is a list containing all the DE standard forms.Example 3:Let be DE: $\ {\partial y \over \partial t} = {\partial ^2y \over \partial x^2} - e^{-t}(sin(πx) - π^2sin(πx))$Standard form of DE: $\ {\partial y \over \partial t} - {\partial ^2y \over \partial x^2} + e^{-t}(sin(πx) - π^2sin(πx)) = 0$def pde(geom,y):
	x = geom.get(0)
	t = geom.get(1)
	dydt = pinns.Grad.jacobian(y, geom, xi=1)  
	dydxx = pinns.Grad.hessian(y, geom, xi=0, xj=0)
	return dydt - dydxx + tf.math.exp(-t) * (tf.math.sin(np.pi*x) - np.pi**2*tf.math.sin(np.pi*x))Note 1:In this mathematical formula, the second independent variable was named $x$. So if the argument for the input vector was also $x$ there would be confusion. For this, the name changed togeom, fromgeometry. If you do this, however, you should not forget to havegeomas an argument insidejacobian()andhessian(), and not $x$.Note 2:It is important to note that inside the derivative methods, $y$ and $x$ are fed whole, as they were taken as arguments. Any specification for the type of gradient is given from the index parameters.Wrong:def pde(x,y):
	x1 = x.get(1)
	dydx1 = pinns.Grad.jacobian(y,x1)Right:def pde(x,y):
	dydx1 = pinns.Grad.jacobian(y,x,xi=1)Define the initial conditionsInitial conditions are equations which include y and/or its derivatives, and use a subset or a specific point of domain x. If there are derivatives, the order of condition must be smaller than the order of the DE.
Initial conditions are used as part of the loss function, since they are essential for the NN to converge to a specific solution. Without them, the solution of the DE gives us a family of solutions, not a specific one, so the NN cannot work.
They are handled by theinitial_conditionsmodule.Comment:Technically there is a difference between the terms initial condition and boundary condition, the former describing time conditions and the latter space conditions. However, for convenience we are going to call them all initial conditions.Transformation of initial conditionsTo declare an initial condition, first you have to transform it to its normal form, meaning the left side of the equation contains only the highest order derivative, or the y function if there is no derivative.Example 1:Given form: $y(0) - 1 = 0$Normal form: $y(0) = 1$Example 2:Given form: ${{{\partial^2y_1} \over \partial x_0^2}+2x_0} = {\partial y_0 \over \partial x_0}$ , with $x \in [x_0,0]$Normal form: ${{\partial^2y_1} \over \partial x_0^2} = {\partial y_0 \over \partial x_0} -2x_0$ , with $x \in [x_0,0]$Creation of initial conditionsInitial conditions are created with theICclass. Its constructor takes 5 parameters:x_ic(number/list):Domain of initial conditionf(number/function):Function or number on the right side of the equationy_der(int):Order of y on the left side of the equationder_i(list):Indices of domain variables with respect to which y is derivedyi(int):Index of dependent variable that is being differentiatedConditions likeExample 1are very easy to create, using only the first 2 arguments.Example 1:ic1 = pinns.IC(x_ic=0, f=1)If the right side is a function, we have to declare a method representing this function, according to the guidelines specified at sectionCreate the loss function method. For the previousExample 2, the right side function is declared as:def y_out(x_ic,y):
	dy0x0 = pinns.Grad.jacobian(y, x_ic, yi=0, xi=0)
	x0 = x_ic.get(0)
	return dy0x0 - 2*x0The last three arguments are meant for the specification of the left side of the equation. ForExample 2, they would bey_der=2,der_i=[0,0]andyi=1.Example 2:ic2 = pinns.IC(x_ic=[x.get(0),0], f=y_out, y_der=2, der_i=[0,0], yi=1)When the DE is ordinary and there is a derivative on the left side, theder_ican be skipped, since there is only one independent variable with respect of which the differentiation can happen.Note 1:All the dependent variables and their derivatives in the condition are created by the same domainx_ic. This means that for now periodic conditions like $y(0) = y(1)$, are not supported.Note 2:For now, degrees larger than 1 on the left side of the equation are not supported. This means that the dependent variable of the highest order must not be raised to a power,e.g.${y'(0)}^2 =1$.Build and train your modelModel creation and training are handled by thetrainingmodule. It is based on and utilizes properties of the Keras library. The model does not belong to apinnsclass, so the user can inspect it or train it further, taking advantage of all Keras utilities.Build the neural networkThe neural network is created with thenet()method. It returns a sequential NN based on the Keras library. It has 4 parameters:inputs(int):Number of input nodeslayers(list):Nodes per hidden layeractivation(string/list):Activation function/functions for hidden layersoutputs(int):Number of output nodesThe input layer of the NN must have as many nodes as independent variables, and the output layer as many nodes as dependent variables. The number of hidden layers and the number of nodes in each layer is up to the user. Usually the more complicated the DE is, the more nodes and layers it requires.The activation function for each layer is also up to the user, although empirically,tanhfunction works best for DEs. The available activation function strings are the same as those provided by Keras.Example 1:Let be partial DE with 3 independent variables. We want 2 hidden layers with 24 nodes each, and the activation function for both of them to betanh:model = pinns.net(inputs=3, layers=[24,24], activation='tanh', outputs=1)Example 2:Let be a system of 2 ordinary DEs. We want 3 hidden layers with 12,24, and 12 nodes, and activation functionsigmoid,tanh, andsigmoidrespectfully:model = pinns.net(inputs=1,layers=[12,24,12],activation=['sigmoid','tanh','sigmoid'], outputs=2)Train the modelThe training of a model is executed by thetrain()method. It does not return something, but it changes the model's weights, so it can solve the DE. It has 6 parameters:model(keras.engine.sequential.Sequential):Neural network modelx_domain(pinns.Domain):Domain of DEpde(method):Loss function that returns the standard form of the DEic_list(list):Initial conditions of the DEepochs(int):Number of training epochslr(float):Learning rate of trainingExample:pinns.train(model, x, pde, [ic1,ic2], epochs=3000, lr=0.01)Note: Even if there is one initial condition, it has to be put into a list.While executed,train()prints the total loss, the pde loss and the initial conditions loss every 1000 epochs. At the end it prints the total time needed for the training.Complete implementation exampleLet be DE:$y^{(3)} = 2$With initial conditions:$y''(1) = 2y'(1)-5$$y'(-1)=1$$y(-1)=0$import pinns

x = pinns.Domain(-1, 1, 100)  

def pde(x, y):  
    dy_xx = pinns.Grad.hessian(y, x)  
    dy_xxx = pinns.Grad.jacobian(dy_xx, x)  
    return dy_xxx - 2  

def ic_out(x_in, y):  
    dyx = pinns.Grad.jacobian(y, x_in)  
    return 2*dyx-5  

ic1 = pinns.IC(x_ic=1, f=ic_out, y_der=2)  
ic2 = pinns.IC(x_ic=-1, f=1, y_der=1)  
ic3 = pinns.IC(x_ic=-1, f=0, y_der=0)

model = pinns.net(inputs=1, layers=3 * [60], activation='tanh', outputs=1)  
pinns.train(model, x, pde, [ic1, ic2, ic3], epochs=2000, lr=0.001)Output:Training starts...
Epoch: 0		Total Loss = 2.60e+01	PDE Loss = 4.45e+00		BC Loss = 2.15e+01
Epoch: 1000		Total Loss = 1.18e-04	PDE Loss = 1.18e-04		BC Loss = 3.20e-08
Epoch: 2000		Total Loss = 4.24e-04	PDE Loss = 6.71e-05		BC Loss = 3.57e-04
Training took 140.820704959 sNote: The model chosen after the training is the one which corresponds to the last epoch, not the one with the smallest total loss. This is meant to be fixed in a later version.Print the resultsYou can use thematplotliblibrary to print the graph of the approximate solution, and if you know the real solution, you can overlap and compare them. For the example above, the real solution is:$y(x) = {1 \over 3}x^3+{5 \over 6}x^2+{5 \over 3}x+{7 \over 6}$We type:import matplotlib.pyplot as plt  
import numpy as np  
  
x_test = np.linspace(-1, 1, 100)  
y_true = 1/3*x_test**3 + 5/6*x_test**2 + 5/3*x_test + 7/6  
y_pred = model(x_test)  
  
plt.plot(y_true)  
plt.plot(y_pred)  
plt.title('Evaluation')  
plt.legend(['Real', 'Predicted'])  
plt.show()As we can see, the training was successful, since the two graphs coincide perfectly.LicenseGNU General Public License v3.0 or later.Open LICENSE to see the full text.AuthorAlexandros MavromatisEmail:alex38clues@gmail.comGithub:https://github.com/alexandmi