Announcement: PyTorch - Blender integration

username: Christoph_Heindl
cooked: <p>Training with artificial images is becoming increasingly important to address the lack of real data sets in various niche areas. Yet, many today’s approaches write 2D/3D simulations from scratch.</p>
<p>To improve this situation and make better use of existing pipelines, we’ve been working towards an integration between Blender, an open-source real-time physics enabled animation software, and PyTorch. Today we announce <strong>blendtorch</strong>, an open-source Python library that seamlessly integrates distributed Blender renderings into PyTorch data pipelines at 60FPS (640x480 RGBA).</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/cheind/pytorch-blender/" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/38565?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/cheind/pytorch-blender/" target="_blank" rel="nofollow noopener">cheind/pytorch-blender</a></h3>

<p>Seamless integration of Blender into PyTorch datasets. - cheind/pytorch-blender</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>– Christoph</p>

username: tom
cooked: <p>That looks very cool. Thank you for sharing!</p>
<p>Thomas</p>

username: ptrblck
cooked: <p>This looks really cool! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Christoph_Heindl
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/tom">@tom</a> thanks for your feedback. We have just released v0.2 which brings an even better PyTorch integration, plus OpenAI.gym support that allows us to use Blender as modelling, simulation and interactive manipulation framework to train your RL agents through the well known OpenAI gym interface. Below is an animated gif showing a recreation of the cartpole demo running in Blender.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/5309bc54c5d09e9655cf2ac5102fa21bf0e87989.gif" data-download-href="https://discuss.pytorch.org/uploads/default/5309bc54c5d09e9655cf2ac5102fa21bf0e87989" title="capture|500x"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/5309bc54c5d09e9655cf2ac5102fa21bf0e87989_2_690x407.gif" alt="capture|500x" data-base62-sha1="bQAuig2xdM3et0EHIfB1Oem1WRP" width="690" height="407" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/5309bc54c5d09e9655cf2ac5102fa21bf0e87989_2_690x407.gif, https://discuss.pytorch.org/uploads/default/original/3X/5/3/5309bc54c5d09e9655cf2ac5102fa21bf0e87989.gif 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/3/5309bc54c5d09e9655cf2ac5102fa21bf0e87989.gif 2x" data-dominant-color="4D4D4D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">capture|500x</span><span class="informations">727×429 1.41 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/cheind/pytorch-blender">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/cheind/pytorch-blender" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/17e7032500be48548847c493a7bd931d197f8600fb865a31d435d05f693d837b/cheind/pytorch-blender" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/cheind/pytorch-blender" target="_blank" rel="noopener nofollow ugc">GitHub - cheind/pytorch-blender: :sweat_drops: Seamless, distributed,...</a></h3>

  <p>:sweat_drops: Seamless, distributed, real-time integration of Blender into PyTorch data pipelines - GitHub - cheind/pytorch-blender: :sweat_drops: Seamless, distributed, real-time integration of Bl...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Devendra_Sharma
cooked: <p>Hi <a class="mention" href="/u/christoph_heindl">@Christoph_Heindl</a> !</p>
<p>Your project looks really interesting…</p>
<p>I am trying to create a chatbot using Blender and Pytorch. I’ve got the blend file in which I’ve created the poses (the rain rig). The Rain rig responds to the audio input based on the (.dat) file. And honestly,  I’m still clueless about how it gonna create poses based on output from the conversational module in <em>real-time</em>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b77ef11ab8c79a4ca3ca450b88a6d2c56cee8f79.png" data-download-href="https://discuss.pytorch.org/uploads/default/b77ef11ab8c79a4ca3ca450b88a6d2c56cee8f79" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b77ef11ab8c79a4ca3ca450b88a6d2c56cee8f79_2_690x379.png" alt="image" data-base62-sha1="qbhlCiR8v2K2EnmMUUFICb1HSnT" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b77ef11ab8c79a4ca3ca450b88a6d2c56cee8f79_2_690x379.png, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b77ef11ab8c79a4ca3ca450b88a6d2c56cee8f79.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b77ef11ab8c79a4ca3ca450b88a6d2c56cee8f79.png 2x" data-dominant-color="F7F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">879×484 95.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Although I came across <strong>blendtorch</strong>- a Python framework to seamlessly integrate Blender into PyTorch datasets for deep learning from artificial visual data</p>
<p>But I’m still wondering how it will create poses on the go ie. based on conversational input in real-time. Can you please guide me a little on how shall I proceed?</p>
<p>I’m trying to create something like this:</p>
<ol>
<li><a href="https://github.com/huihut/Facemoji" class="inline-onebox" rel="noopener nofollow ugc">GitHub - huihut/Facemoji: 😆 A voice chatbot that can imitate your expression. OpenCV+Dlib+Live2D+Moments Recorder+Turing Robot+Iflytek IAT+Iflytek TTS</a> or just like digital human Fatema who answers customers’ queries.</li>
</ol>
<p>Thanks in advance.</p>

username: God_Bennett
cooked: <p>Hey <a class="mention" href="/u/devendra_sharma">@Devendra_Sharma</a> , I was also working on a chatbot in Blender as well, for about a month now.</p>
<p>The chatbot’s simple 3d mechanics are done, just need to integrate the conversational ai component now.</p>
<p>Any luck importing any torch conversation lib so far into blender?</p>

username: ClintochX
cooked: <p>This BlendTorch project looks promising. I love It.</p>
<p><span class="hashtag">#b3d</span> <span class="hashtag">#ai</span> <span class="hashtag">#NextGen</span></p>