Package: resampler
Hugues HoppeAug 2022.[Open in Colab][Kaggle][MyBinder][DeepNote][GitHub source][API docs][PyPI package]The notebookresampler_notebook.ipynbdemonstrates theresampler libraryand contains documentation, usage examples, unit tests, and experiments.OverviewTheresamplerlibrary enables fast differentiable resizing and warping of arbitrary grids.
It supports:grids ofany dimension(e.g., 1D, 2D images, 3D video, 4D batches of videos), containingsamples of any shape(e.g., scalars, colors, motion vectors, Jacobian matrices) andanynumeric type(e.g.,uint8,float64,complex128)within severalarray libraries(numpy,tensorflow,torch, andjax);either'dual'("half-integer") or'primal'grid-typefor each dimension;manyboundaryrules, specified per dimension, extensible via subclassing;an extensible set offilterkernels, selectable per dimension;optionalgammatransfer functions for correct linear-space filtering;prefiltering for accurateantialiasingwhenresizedownsampling;efficient backpropagation ofgradientsfortensorflow,torch, andjax;few dependencies (onlynumpyandscipy) andno C extension code, yetfaster resizingthan C++ implementations
intf.imageandtorch.nn.A key strategy is to leverage existing sparse matrix representations and operations.Example usage!pipinstall-qmediapynumpyresamplerimportmediapyasmediaimportnumpyasnpimportresamplerarray=np.random.default_rng(1).random((4,6,3))# 4x6 RGB image.upsampled=resampler.resize(array,(128,192))# To 128x192 resolution.media.show_images({'4x6':array,'128x192':upsampled},height=128)image=media.read_image('https://github.com/hhoppe/data/raw/main/image.png')downsampled=resampler.resize(image,(32,32))media.show_images({'128x128':image,'32x32':downsampled},height=128)importmatplotlib.pyplotaspltarray=[3.0,5.0,8.0,7.0]# 4 source samples in 1D.new_dual=resampler.resize(array,(32,))# (default gridtype='dual') 8x resolution.new_primal=resampler.resize(array,(25,),gridtype='primal')# 8x resolution._,axs=plt.subplots(1,2,figsize=(7,1.5))axs[0].set_title("gridtype='dual'")axs[0].plot((np.arange(len(array))+0.5)/len(array),array,'o')axs[0].plot((np.arange(len(new_dual))+0.5)/len(new_dual),new_dual,'.')axs[1].set_title("gridtype='primal'")axs[1].plot(np.arange(len(array))/(len(array)-1),array,'o')axs[1].plot(np.arange(len(new_primal))/(len(new_primal)-1),new_primal,'.')plt.show()batch_size=4batch_of_images=media.moving_circle((16,16),batch_size)upsampled=resampler.resize(batch_of_images,(batch_size,64,64))media.show_videos({'original':batch_of_images,'upsampled':upsampled},fps=1)originalupsampledMost examples above use the defaultresize()settings:gridtype='dual'for both source and destination arrays,boundary='auto'which uses'reflect'for upsampling and'clamp'for downsampling,filter='lanczos3'(aLanczoskernel with radius 3),gamma=Nonewhich by default uses the'power2'transfer function for theuint8image in the second example,scale=1.0, translate=0.0(no domain transformation),defaultprecisionand outputdtype.Advanced usage:Map an image to a wider grid using customscaleandtranslatevectors,
with horizontal'reflect'and vertical'natural'boundary rules,
providing a constant value for the exterior,
using different filters (Lanczos and O-MOMS) in the two dimensions,
disabling gamma correction, performing computations in double-precision,
and returning an output array in single-precision:new=resampler.resize(image,(128,512),boundary=('natural','reflect'),cval=(0.2,0.7,0.3),filter=('lanczos3','omoms5'),gamma='identity',scale=(0.8,0.25),translate=(0.1,0.35),precision='float64',dtype='float32')media.show_images({'image':image,'new':new})Warp an image by transforming it usingpolar coordinates:shape=image.shape[:2]yx=((np.indices(shape).T+0.5)/shape-0.5).T# [-0.5, 0.5]^2radius,angle=np.linalg.norm(yx,axis=0),np.arctan2(*yx)angle+=(0.8-radius).clip(0,1)*2.0-0.6coords=np.dstack((np.sin(angle)*radius,np.cos(angle)*radius))+0.5resampled=resampler.resample(image,coords,boundary='constant')media.show_images({'image':image,'resampled':resampled})Limitations:Filters are assumed to beseparable.Althoughresizeimplements prefiltering,resampledoes not yet have it (and therefore
may have aliased results if downsampling).Differentiability is only with respect to the grid values,
not wrt the resize shape, scale, translation, or the resampling coordinates.