How to extract the landmark from the incomplete face image?

FesianXu: <p>I am doing some researches on lip reading and need to extract the landmarks of the mouth. However, the available tech I know like face++ <a href="https://console.faceplusplus.com/documents/5679127" class="inline-onebox" rel="noopener nofollow ugc">Face⁺⁺</a> can only get the landmark of the complete face or It will complain that cannot detect the human face. My samples look like, which are incomplete human face, only the area of mouth are available.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/5735a8edee5e9fb777829f0852a687d476c86bc4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5735a8edee5e9fb777829f0852a687d476c86bc4" title="QQ%E6%88%AA%E5%9B%BE20191002090416"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5735a8edee5e9fb777829f0852a687d476c86bc4_2_690x125.jpeg" alt="QQ%E6%88%AA%E5%9B%BE20191002090416" data-base62-sha1="cruvlSMlUfcBctok8CRR2ZgluYc" width="690" height="125" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5735a8edee5e9fb777829f0852a687d476c86bc4_2_690x125.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5735a8edee5e9fb777829f0852a687d476c86bc4_2_1035x187.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5735a8edee5e9fb777829f0852a687d476c86bc4_2_1380x250.jpeg 2x" data-dominant-color="AF9185"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ%E6%88%AA%E5%9B%BE20191002090416</span><span class="informations">1695×309 39 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What can i do to extract the landmarks of the mouth in such a incomplete human face images? Thanks for any advises.<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

alex.veuthey: <p>One idea would be to retrain some model like MTCNN on available face / landmarks databases (with more landmarks than the original MTCNN of course), but this time cropping the image to your kind of input and discarding the landmarks you wouldn’t use. It would take minimal code adjustment but heavy preprocessing of the training images…</p>

FesianXu: <p>thanks for your advise<br>
and btw, i found a wonderful project that can do 3d face alignment even the face is incomplete<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/1adrianb/face-alignment" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/2567045?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/1adrianb/face-alignment" target="_blank" rel="nofollow noopener">1adrianb/face-alignment</a></h3>

<p>:fire: 2D and 3D Face alignment library build using pytorch  - 1adrianb/face-alignment</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>