Package: torchfields
torchfieldsAPyTorchadd-on for working with image mappings and displacement fields, including Spatial TransformersTorchfields provides an abstraction that neatly encapsulates the functionality of displacement fields
as used inSpatial Transformer NetworksandOptical Flow Estimation.Fields can be treated as normal PyTorch tensors for most
purposes, and also include additional functionality for composing
displacements and sampling from tensors.InstallationTo install torchfields simply dopip install torchfieldsIntroductionAdisplacement fieldrepresents amappingorflowthat indicates how an image should be warped.It is essentially a spatial tensor containing displacement vectors at each pixel, where each displacement vector indicates the displacement distance and direction at that pixel.Displacement field conventionsUnitsThe standard unit of displacement is ahalf-image, so a displacement vector of magnitude 2 means that the displacement distance is equal to the side length of the displaced image.Note:This convention originates from the originalSpatial Transformer Networkspaper where such fields were presented as mappings, with -1 representing the left or top edge of the image, and +1 representing the right or bottom edge.torchfieldsalso supports seamlessly converting to and from units ofpixelsusing thepixels()andfrom_pixels()functions.Displacement directionThe most common way to warp an image by a displacement field is by sampling from it at the points pointed to by the field vectors.
This is often referred to as theEulerianorpullconvention, since the vectors in the field point to the locations from which the image should bepulled.
This is achieved by calling thesample()function (which in fact wraps PyTorch's built-ingrid_sample(), while converting the conventions as necessary).An alternative way to warp an image by a displacement field is by sending each pixel of the image along the corresponding displacement vector to its new location. This is referred to as theLagrangianorpushconvention, since the vectors of the field indicate where an image pixel should bepushedto. This direction, while seemingly intuitive, is much less straight-forward to implement, since there is no definitive way to handle the discretization (for instance, what to do when the destinations are not whole pixel coordinates, when two sources map to the same destination, and when nothing maps into a destination pixel).
The solution for warping in the Lagrangian direction is tofirst invert the fieldusinginverse(), and then warp the image normally usingsample().To read more about the two ways to describe flow fields, see theWikipedia articleon the subject.Relationship to PyTorch tensorsDisplacement fields inherit fromtorch.Tensor, so all functionality fromPyTorchtensors also works with displacement fields. That is, any PyTorch function that accepts atorch.Tensortype will also implicitly accept atorchfieldsdisplacement field.Furthermore, the module installs itself (through monkey patching) astorch.Fieldmirroring thetorch.Tensormodule, and all the functionality of thetorchfieldspackage can be conveniently accessed through that shortcut. This shortcut gets activated at the first import (usingimport torchfields).Note, however, that thetorchfieldspackage is neither endorsed by nor maintained by the PyTorch developer community, and is instead a separate project maintained by researchers at Princeton University.TutorialTo learn more and get started with usingtorchfieldscheck out thetutorial.