Package: dedeshot
D3DShotD3DShotis a pure Python implementation of theWindows Desktop Duplication API. It leverages DXGI and Direct3D system libraries to enable extremely fast and robust screen capture functionality for your Python scripts and applications on Windows.D3DShot:Is by far the fastest way to capture the screen with Python on Windows 8.1+Is very easy to use. If you can remember 10-ish methods, you know the entire thing.Covers all common scenarios and use cases:Screenshot to memoryScreenshot to diskScreenshot to memory buffer every X seconds (threaded; non-blocking)Screenshot to disk every X seconds (threaded; non-blocking)High-speed capture to memory buffer (threaded; non-blocking)Captures to PIL Images out of the box. Gracefully adds output options if NumPy or PyTorch can be found.Detects displays in just about any configuration: Single monitor, multiple monitors on one adapter, multiple monitors on multiple adapters.Handles display rotation and scaling for youSupports capturing specific regions of the screenIs robust and very stable. You can run it for hours / days without performance degradationIs even able to capture DirectX 11 / 12 exclusive fullscreen applications and games!TL;DR Quick Code SamplesScreenshot to Memoryimportd3dshotd=d3dshot.create()d.screenshot()Out[1]: <PIL.Image.Image image mode=RGB size=2560x1440 at 0x1AA7ECB5C88>Screenshot to Diskimportd3dshotd=d3dshot.create()d.screenshot_to_disk()Out[1]: './1554298682.5632973.png'Screen Capture for 5 Seconds and Grab the Latest Frameimportd3dshotimporttimed=d3dshot.create()d.capture()time.sleep(5)# Capture is non-blocking so we wait explicitelyd.stop()d.get_latest_frame()Out[1]: <PIL.Image.Image image mode=RGB size=2560x1440 at 0x1AA044BCF60>Screen Capture the Second Monitor as NumPy Arrays for 3 Seconds and Grab the 4 Latest Frames as a Stackimportd3dshotimporttimed=d3dshot.create(capture_output="numpy")d.display=d.displays[1]d.capture()time.sleep(3)# Capture is non-blocking so we wait explicitelyd.stop()frame_stack=d.get_frame_stack((0,1,2,3),stack_dimension="last")frame_stack.shapeOut[1]: (1080, 1920, 3, 4)This is barely scratching the surface... Keep reading!RequirementsWindows 8.1+ (64-bit)Python 3.6+ (64-bit)Installationpip install d3dshotD3DShotleverages DLLs that are already available on your system so the dependencies are very light. Namely:comtypes: Internal use. To preserve developer sanity while working with COM interfaces.Pillow: Default Capture Output. Also used to save to disk as PNG and JPG.These dependencies will automatically be installed alongsideD3DShot; No need to worry about them!Extra Step: Laptop UsersWindows has a quirk when using Desktop Duplication on hybrid-GPU systems. Please see thewiki articlebefore attempting to useD3DShoton your system.ConceptsCapture OutputsThe desiredCapture Outputis defined when creating aD3DShotinstance. It defines the type of all captured images. By default, all captures will return PIL.Image objects. This is a good option if you mostly intend to take screenshots.# Captures will be PIL.Image in RGB moded=d3dshot.create()d=d3dshot.create(capture_output="pil")D3DShotis however quite flexible! As your environment meets certain optional sets of requirements, more options become available.IfNumPyis available# Captures will be np.ndarray of dtype uint8 with values in range (0, 255)d=d3dshot.create(capture_output="numpy")# Captures will be np.ndarray of dtype float64 with normalized values in range (0.0, 1.0)d=d3dshot.create(capture_output="numpy_float")IfNumPyandPyTorchare available# Captures will be torch.Tensor of dtype uint8 with values in range (0, 255)d=d3dshot.create(capture_output="pytorch")# Captures will be torch.Tensor of dtype float64 with normalized values in range (0.0, 1.0)d=d3dshot.create(capture_output="pytorch_float")IfNumPyandPyTorchare available +CUDAis installed andtorch.cuda.is_available()# Captures will be torch.Tensor of dtype uint8 with values in range (0, 255) on device cuda:0d=d3dshot.create(capture_output="pytorch_gpu")# Captures will be torch.Tensor of dtype float64 with normalized values in range (0.0, 1.0) on device cuda:0d=d3dshot.create(capture_output="pytorch_float_gpu")Trying to use a Capture Output for which your environment does not meet the requirements will result in an error.SingletonWindows only allows 1 instance of Desktop Duplication per process. To make sure we fall in line with that limitation to avoid issues, theD3DShotclass acts as a singleton. Any subsequent calls tod3dshot.create()will always return the existing instance.d=d3dshot.create(capture_output="numpy")# Attempting to create a second instanced2=d3dshot.create(capture_output="pil")# Only 1 instance of D3DShot is allowed per process! Returning the existing instance...# Capture output remains 'numpy'd2.capture_output.backend# Out[1]: <d3dshot.capture_outputs.numpy_capture_output.NumpyCaptureOutput at 0x2672be3b8e0>d==d2# Out[2]: TrueFrame BufferWhen you create aD3DShotinstance, a frame buffer is also initialized. It is meant as a thread-safe, first-in, first-out way to hold a certain quantity of captures and is implemented as acollections.deque.By default, the size of the frame buffer is set to 60. You can customize it when creating yourD3DShotobject.d=d3dshot.create(frame_buffer_size=100)Be mindful of RAM usage with larger values; You will be dealing with uncompressed images which use up to 100 MB each depending on the resolution.The frame buffer can be accessed directly withd.frame_bufferbut the usage of the utility methods instead is recommended.The buffer is used by the following methods:d.capture()d.screenshot_every()It is always automatically cleared before starting one of these operations.DisplaysWhen you create aD3DShotinstance, your available displays will automatically be detected along with all their relevant properties.d.displaysOut[1]: 
[<Display name=BenQ XL2730Z (DisplayPort) adapter=NVIDIA GeForce GTX 1080 Ti resolution=2560x1440 rotation=0 scale_factor=1.0 primary=True>,
 <Display name=BenQ XL2430T (HDMI) adapter=Intel(R) UHD Graphics 630 resolution=1920x1080 rotation=0 scale_factor=1.0 primary=False>]By default, your primary display will be selected. At all times you can verify which display is set to be used for capture.d.displayOut[1]: <Display name=BenQ XL2730Z (DisplayPort) adapter=NVIDIA GeForce GTX 1080 Ti resolution=2560x1440 rotation=0 scale_factor=1.0 primary=True>Selecting another display for capture is as simple as settingd.displayto another value fromd.displaysd.display=d.displays[1]d.displayOut[1]: <Display name=BenQ XL2430T (HDMI) adapter=Intel(R) UHD Graphics 630 resolution=1080x1920 rotation=90 scale_factor=1.0 primary=False>Display rotation and scaling is detected and handled for you byD3DShot:Captures on rotated displays will always be in the correct orientation (i.e. matching what you see on your physical displays)Captures on scaled displays will always be in full, non-scaled resolution (e.g. 1280x720 at 200% scaling will yield 2560x1440 captures)RegionsAll capture methods (screenshots included) accept an optionalregionkwarg. The expected value is a 4-length tuple of integers that is to be structured like this:(left, top, right, bottom)  # values represent pixelsFor example, if you want to only capture a 200px by 200px region offset by 100px from both the left and top, you would do:d.screenshot(region=(100,100,300,300))If you are capturing a scaled display, the region will be computed against the full, non-scaled resolution.If you go through the source code, you will notice that the region cropping happens after a full display capture. That might seem sub-optimal but testing has revealed that copying a region of the GPUD3D11Texture2Dto the destination CPUD3D11Texture2DusingCopySubresourceRegionis only faster when the region is very small. In fact, it doesn't take long for larger regions to actually start becoming slower than the full display capture using this method. To make things worse, it adds a lot of complexity by having the surface pitch not match the buffer size and treating rotated displays differently. It was therefore decided that it made more sense to stick toCopyResourcein all cases and crop after the fact.UsageCreate a D3DShot instanceimportd3dshotd=d3dshot.create()createaccepts 2 optional kwargs:capture_output: Which capture output to use. See theCapture Outputssection underConceptsframe_buffer_size: The maximum size the frame buffer can grow to. See theFrame Buffersection underConceptsDo NOT import theD3DShotclass directly and attempt to initialize it yourself! Thecreatehelper function initializes and validates a bunch of things for you behind the scenes.Once you have aD3DShotinstance in scope, we can start doing stuff with it!List the detected displaysd.displaysSelect a display for captureYour primary display is selected by default but if you have a multi-monitor setup, you can select another entry ind.displaysd.display=d.displays[1]Take a screenshotd.screenshot()screenshotaccepts 1 optional kwarg:region: A region tuple. See theRegionssection underConceptsReturns: A screenshot with a format that matches the capture output you selected when creating yourD3DShotobjectTake a screenshot and save it to diskd.screenshot_to_disk()screenshot_to_diskaccepts 3 optional kwargs:directory: The path / directory where to write the file. If omitted, the working directory of the program will be usedfile_name: The file name to use. Permitted extensions are:.png,.jpg. If omitted, the file name will be<time.time()>.pngregion: A region tuple. See theRegionssection underConceptsReturns: A string representing the full path to the saved image fileTake a screenshot every X secondsd.screenshot_every(X)# Where X is a number representing secondsThis operation is threaded and non-blocking. It will keep running untild.stop()is called. Captures are pushed to the frame buffer.screenshot_everyaccepts 1 optional kwarg:region: A region tuple. See theRegionssection underConceptsReturns: A boolean indicating whether or not the capture thread was startedTake a screenshot every X seconds and save it to diskd.screenshot_to_disk_every(X)# Where X is a number representing secondsThis operation is threaded and non-blocking. It will keep running untild.stop()is called.screenshot_to_disk_everyaccepts 2 optional kwargs:directory: The path / directory where to write the file. If omitted, the working directory of the program will be usedregion: A region tuple. See theRegionssection underConceptsReturns: A boolean indicating whether or not the capture thread was startedStart a high-speed screen captured.capture()This operation is threaded and non-blocking. It will keep running untild.stop()is called. Captures are pushed to the frame buffer.captureaccepts 2 optional kwargs:target_fps: How many captures per second to aim for. The effective capture rate will go under if the system can't keep up but it will never go over this target. It is recommended to set this to a reasonable value for your use case in order not to waste system resources. Default is set to 60.region: A region tuple. See theRegionssection underConceptsReturns: A boolean indicating whether or not the capture thread was startedGrab the latest frame from the bufferd.get_latest_frame()Returns: A frame with a format that matches the capture output you selected when creating yourD3DShotobjectGrab a specific frame from the bufferd.get_frame(X)# Where X is the index of the desired frame. Needs to be < len(d.frame_buffer)Returns: A frame with a format that matches the capture output you selected when creating yourD3DShotobjectGrab specific frames from the bufferd.get_frames([X,Y,Z,...])# Where X, Y, Z are valid indices to desired framesReturns: A list of frames with a format that matches the capture output you selected when creating yourD3DShotobjectGrab specific frames from the buffer as a stackd.get_frame_stack([X,Y,Z,...],stack_dimension="first|last")# Where X, Y, Z are valid indices to desired framesOnly has an effect on NumPy and PyTorch capture outputs.get_frame_stackaccepts 1 optional kwarg:stack_dimension: One offirst,last. Which axis / dimension to perform the stack onReturns: A single array stacked on the specified dimension with a format that matches the capture output you selected when creating yourD3DShotobject. If the capture output is not stackable, returns a list of frames.Dump the frame buffer to diskThe files will be named according to this convention:<frame buffer index>.pngd.frame_buffer_to_disk()frame_buffer_to_diskaccepts 1 optional kwarg:directory: The path / directory where to write the files. If omitted, the working directory of the program will be usedReturns: NonePerformanceMeasuring the exact performance of the Windows Desktop Duplication API proves to be a little complicated because it will only return new texture data if the contents of the screen has changed. This is optimal for performance but it makes it difficult to express in terms of frames per second, the measurement people tend to expect for benchmarks. Ultimately the solution ended up being to run a high FPS video game on the display to capture to make sure the screen contents is different at all times while benchmarking.As always, remember that benchmarks are inherently flawed and highly depend on your individual hardware configuration and other circumstances. Use the numbers below as a relative indication of what to expect fromD3DShot, not as some sort of absolute truth.2560x1440 onNVIDIA GTX 1080 Ti1920x1080 onIntel UHD Graphics 6301080x1920 (vertical) onIntel UHD Graphics 630"pil"29.717 FPS47.75 FPS35.95 FPS"numpy"57.667 FPS58.1 FPS58.033 FPS"numpy_float"18.783 FPS29.05 FPS27.517 FPS"pytorch"57.867 FPS58.1 FPS34.817 FPS"pytorch_float"18.767 FPS28.367 FPS27.017 FPS"pytorch_gpu"27.333 FPS35.767 FPS34.8 FPS"pytorch_float_gpu"27.267 FPS37.383 FPS35.033 FPSThe absolute fastest capture outputs appear to be"numpy"and unrotated"pytorch"; all averaging around 58 FPS. In Python land, this is FAST!How is the "numpy" capture output performancethatgood?NumPy arrays have a ctypes interface that can give you their raw memory address (X.ctypes.data). If you have the memory address and size of another byte buffer, which is what we end up with by processing what returns from the Desktop Duplication API, you can usectypes.memmoveto copy that byte buffer directly to the NumPy structure, effectively bypassing as much Python as possible.In practice it ends up looking like this:ctypes.memmove(np.empty((size,),dtype=np.uint8).ctypes.data,pointer,size)This low-level operation is extremely fast, leaving everything else that would normally compete with NumPy in the dust.Why is the "pytorch" capture output slower on rotated displays?Don't tell anyone but the reason it can compete with NumPy in the first place is only because...it isgenerated from a NumPy array built from the method above! If you sniff around the code, you will indeed findtorch.from_numpy()scattered around. This pretty much matches the speed of the "numpy" capture output 1:1, except when dealing with a rotated display. Display rotation is handled bynp.rot90()calls which yields negative strides on that array. Negative strides are understood and perform well under NumPy but are still unsupported in PyTorch at the time of writing. To address this, an additional copy operation is needed to bring it back to a contiguous array which imposes a performance penalty.Why is the "pil" capture output, being the default, not the fastest?PIL has no ctypes interface like NumPy so a bytearray needs to be read into Python first and then fed toPIL.Image.frombytes(). This is still fast in Python terms, but it just cannot match the speed of the low-level NumPy method.It remains the default capture output because:PIL Image objects tend to be familiar to Python usersIt's a way lighter / simpler dependency for a library compared to NumPy or PyTorchWhy are the float versions of capture outputs slower?The data of the Direct3D textures made accessible by the Desktop Duplication API is formatted as bytes. To represent this data as normalized floats instead, a type cast and element-wise division needs to be performed on the array holding those bytes. This imposes a major performance penalty. Interestingly, you can see this performance penalty mitigated on GPU PyTorch tensors since the element-wise division can be massively parallelized on the device.Crafted with ❤ by Serpent.AI 🐍Twitter-Twitch