Package: pysmtb
pysmtbpython toolbox of image and rendering related helper / visualization functionsContentsInstallationInteractive image viewerDeferred rendering using PyEmbreeInteractive Sliding ComparisonsInstallationpysmtb is available on pypi.org and can be installed via pip in most environments:pipinstallpysmtbAlternatively, this Git repository can be cloned and an environment can be set up manually using conda:conda env create -f environment.yml
conda activate pysmtbInteractive image viewerpysmtb.ivprovides an interactive (HDR) image viewer with automatic tonemapping:fromglobimportglobfrompysmtb.ivimportivfrompysmtb.utilsimportread_exrfns=glob('*.exr')ims=[read_exr(fn)[0]forfninfns]# by default, the viewer shows the first image and adjusts scale & offset to fit most of the dynamic range into the display rangeiv(ims)# tonemapping can be controllediv(ims,autoscale=False,scale=10,gamma=2)# viewer can automatically arrange multiple images in a "collage"iv(ims,collage=True)# add labels onto each imageiv(ims,labels=fns,annotate=True,annotate_numbers=False)Collage mode can be further controlled, e.g., to pack images of different sizes more densely:# test tight collage modeims1=[np.random.rand(25,15,3)for_inrange(10)]ims2=[np.random.rand(10,12,3)for_inrange(10)]ims3=[np.random.rand(15,12,1)for_inrange(8)]coll=collage(ims1+ims2+ims3,bw=1,tight=False)coll_tight=collage(ims1+ims2+ims3,bw=1,tight=True)iv.iv(dict(tight=coll_tight,non_tight=coll),collage=True,collageBorderWidth=1,collageBorderValue=1,annotate=True)Deferred rendering using PyEmbreeUnderpysmtb.geometry, several utility functions for geometry generation and access are provided, along with functions that call PyEmbree to ray trace a scene, providing numpy ndarrays with, e.g., the 3D intersection points, light and view directions and so on.importnumpyasnpfrompysmtb.renderingimportembree_render_deferredfrompysmtb.ivimportivfrompysmtb.utilsimportDct,assign_masked# create simple test geometrysphere=create_unit_sphere(40)cube=create_unit_cube()# shift cube to the rightcube.vertices+=np.r_[1.5,0.,0.][None]# visualize "scene"sh=scatter3(sphere['vertices'],'.')ax=sh.axesscatter3(cube['vertices'],'.',axes=ax)visualization:3D scatter plot of scene objects# "scene" is represented simply as list of dictionaries, each with vertices, faces and uvs (texture coordinates)meshes=[]meshes.append(sphere)meshes.append(cube)# set camera resolutioncam=Dct(res_x=384,res_y=256)# render scene withbuffers=embree_render_deferred(meshes,with_tangent_frames=True,cam=cam,auto_cam=True,light_position=np.r_[0.,10.,0.])view_dirs_local=assign_masked(buffers['hit'],buffers['view_dirs_local'])light_dirs_local=assign_masked(buffers['hit'],buffers['light_dirs_local'])# visualize resulting local light and view directionsiv(view_dirs_local,light_dirs_local)visualization:view and light vectors in tangent spaceWe can manually call the individual steps (render a few times under randomized camera positions):frompysmtb.renderingimportnormalize,create_unit_sphere,create_unit_cube,embree_intersect_scene,interpolate_vertex_attributes,get_local_dirs,get_bboxbuffers=[]foriinrange(16):# manually render for a bunch of random camera positionscam=Dct(res_x=128,res_y=96)cam.cx=cam.res_x/2.cam.cy=cam.res_y/2.cam.position=np.mean(bbox,axis=0)+(bbox_diam/2+10.*np.random.rand())*normalize(np.random.rand(3)-0.5)# set up EmbreeScenescene,cam=embree_create_scene(meshes=meshes,cam=cam,auto_cam=True,auto_cam_bbox=True,auto_cam_visualize=False)# trace raysbuffers.append(embree_intersect_scene(scene=scene,cam=cam))# get per pixel interpolated vertex attributes (texture coordinates & tangent frames)buffers[-1]=interpolate_vertex_attributes(buffers[-1],meshes)# also compute global & local view directionsview_dirs=get_local_dirs(buffers[-1],cam.position,normalized=True)buffers[-1]['view_dirs_local']=view_dirs['dirs_local']buffers[-1]['view_dirs_global']=view_dirs['dirs_global']# also compute global & local light directionslight_dirs=get_local_dirs(buffers[-1],np.r_[0.,10.,0.],normalized=True)buffers[-1]['light_dirs_local']=light_dirs['dirs_local']buffers[-1]['light_dirs_global']=light_dirs['dirs_global']# num_hits x c --> res_y x res_x x c buffersforkeyinbuffers[-1].keys():ifkeyin['hit']:continuebuffers[-1][key]=assign_masked(buffers[-1]['hit'],buffers[-1][key])visualization:the individual attributeshit maskdepthmesh IDsprimitive (triangle) IDsbarycentric coordinates3D pointsface normalsvertex normalsvertex tangentsvertex bitangentsview dirs (global)view dirs (tangent space)light dirs (global)light dirs (tangent space)texture coordinatesTechnical notesCurrently, meshes are represented as simple dictionaries withvertices,faces,uvsand optionally additional per-vertex attributes (e.g., tangents, bitangents and normals). This is due to a limitation in thetrimesh.Trimeshclass, namely that texture coordinates can only be specified per vertex and not per face, preventing proper unwrapping. Inputs of typetrimesh.Trimeshortrimesh.Sceneare therefore converted to dictionaries or lists of dictionaries with the above keys.The most important function isembree_render_deferred():embree_render_deferred()produces inputs for deferred shading. It takes a list of meshes and a camera dictionary, as well as a point light position as input and returnsnp.ndarraybuffers with all relevant geometric quantities per intersected pixel: 3D intersection points, interpolated vertex normals and tangents, normalized local and unnormalized global light and view directions.For additional light sources,get_local_dirs()can be used with the buffers returned fromembree_render_deferred()and a light position.embree_render_deferred()calls the following functions:embree_create_scene(),embree_intersect_scene(),interpolate_vertex_attributes()andget_local_dirs().embree_create_scene()constructs anEmbreeSceneobject, given the list of meshes. Camera parameters are either user-specified, or will be set automatically (random camera position facing the scene center). Unless explicitly specified, the camera focal length is automatically set so that the scene tightly fits onto the camera sensor. Unless explicitly disabled, it computes per-vertex tangent frames, using each mesh's texture coordinates.embree_intersect_scene()performs the actual ray tracing and returns a pixel mask with the camera's resolution indicating which pixels are hit, as well as buffers for all intersected pixels with the following attributes:
geometry and triangle IDs for each intersection, intersection depth (distance from camera), 3D intersection point,
barycentric coordinates within each triangleinterpolate_vertex_attributes()computes per pixel texture coordinates and tangent frames by interpolating with the
barycentric coordinates returned from theembree_intersect_scene()currently some relevant features are not yet implemented but will be added in the future:tracing shadow rayscamera distortion modelfallback tangent vector computation for meshes without texture coordinatesSliding ComparisonsImages are best compared interactively by flipping back and forth, not statically side-by-side. Below is some simply
HTML, CSS and JavaScript for producing side-by-side views of images or even animations / videos that can be compared
interactively by a sliding divider that can be moved by hovering the mouse over the element.Embedding the necessary JavaScript into websites is trivial:<html><head><metacharset="UTF-8"><title>sliding comparison</title><style>.slider{position:relative;display:flex;}.slider.left{position:absolute;}.slider.right{position:absolute;clip-path:inset(0px0px0px50%);}</style><script>functionslider(event,synced=false){varactiveContainer=event.currentTarget;varactiveRight=activeContainer.querySelector(".right");varoffset=activeRight.getBoundingClientRect().left;if(synced){sliders=document.getElementsByClassName("slider");}else{sliders=[event.currentTarget];}for(vari=0;i<sliders.length;i++){varright=sliders[i].querySelector(".right");varposition=((event.pageX-offset)/right.offsetWidth)*100;right.style.clipPath="inset(0px 0px 0px "+(position)+"%)";}}</script></head><body><divclass="slider"onmousemove="slider(event)"><divclass="left"style="overflow: visible;"><imgsrc="https://raw.githubusercontent.com/smerzbach/data/master/data/mat0386_000.png"/></div><divclass="right"style="overflow: visible;"><imgsrc="https://raw.githubusercontent.com/smerzbach/data/master/data/mat0386_153.png"/></div></div></body></html>Multiple image / video pairs can be arranged in a table-like structure and with a single flag the slider can be moved
globally for all images alike (instead ofonmousemove="slider(event)"just setonmousemove="slider(event, true)".
A slightly more elaborate example HTML document can be found under/data/sliding_comparison.htmlwhich should look and feel something like this: