For details refer to the lecture slides and the original TRPO paper There are many PyTorch implementations of TRPO available - Information from www.telesens.co