Pertained C3D model for video classification

username: Gkv
cooked: <p>Hi all,<br>
I want to extract video features. Is there a pre-trained C3D (<a href="https://arxiv.org/abs/1412.0767" rel="nofollow noopener">https://arxiv.org/abs/1412.0767</a>) network available?.<br>
Thanks</p>

username: zhanghaoinf
cooked: <p>Hi, Gkv,</p>
<p>There are more advanced I3D and P3D pytorch impementations.</p>
<p>P3D: Learning Spatio-Temporal Representation with Pseudo-3D Residual,ICCV 2017<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/qijiezhao/pseudo-3d-pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/20857077?s=400&amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/qijiezhao/pseudo-3d-pytorch" target="_blank" rel="nofollow noopener">qijiezhao/pseudo-3d-pytorch</a></h3>

<p>pseudo-3d-pytorch - pytorch version of pseudo-3d-residual-networks(P-3D),    pretrained model is supported</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I3D: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, CVPR 2017<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/hassony2/kinetics_i3d_pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/10189060?s=400&amp;v=4" class="thumbnail onebox-avatar" width="370" height="370">

<h3><a href="https://github.com/hassony2/kinetics_i3d_pytorch" target="_blank" rel="nofollow noopener">hassony2/kinetics_i3d_pytorch</a></h3>

<p>kinetics_i3d_pytorch - Inflated i3d network with inception backbone, weights transfered from tensorflow</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Gkv
cooked: <p>Hi zhanghaoinf,<br>
Thanks for your kind reply. I am very new to pytorch. Can you provide some links that tell how to use these kinds of implementations in my pytorch code. (I know how to load models using torchvision.models).<br>
Thanks</p>

username: zhanghaoinf
cooked: <p>I have tested P3D-Pytorch. it’s pretty simple and should share similar process with I3D.</p>
<ol>
<li>Pre-process:  For each frame in a clip, there is pre-process like subtracting means, divide std.<br>
An example:</li>
</ol>
<blockquote>
<p>import cv2<br>
mean = (104 / 255.0, 117 / 255.0 ,123 / 255.0)<br>
std     = (0.225, 0.224, 0.229)<br>
frame = cv2.imread(“a string to image path”)<br>
frame /= 255.0 # [0, 255] → [0, 1]<br>
frame -= mean # subtract means<br>
frame /= std      # div std<br>
frame = frame[:, : , (2, 1, 0)] # covert BGR image to RGB image</p>
</blockquote>
<ol start="2">
<li>A clip is a stack of frames with frame size <strong>H x W x 3</strong> and clip size <strong><strong>T x H x W</strong> x 3</strong><br>
since P3D requires input in form of <strong>3 x T x H x W</strong>, perform:</li>
</ol>
<blockquote>
<p>clip = clip.permte(3, 0, 1, 2).contiguous()</p>
</blockquote>
<ol start="3">
<li>Load P3D model<br>
(P3D, Bottleneck, p3d_model_path can be found in the mentioned github.)</li>
</ol>
<blockquote>
<p>model =  P3D(Bottleneck, [3, 8, 36, 3], modality = ‘RGB’)<br>
p3d_weights = torch.load(“a string to p3d model path”)[‘state_dict’]<br>
model.load_state_dict(p3d_weights)<br>
out=model(data)<br>
print(out.size(),out)</p>
</blockquote>

username: Gkv
cooked: <p>Hi,<br>
Thanks a lot for your fast reply. One more doubt, what are these modalities (RGB and Flow)?<br>
Thanks</p>

username: zhanghaoinf
cooked: <p>RGB modality refers to frames which are directly filmed by camera.<br>
Flow refers optical flow that calculated between adjacent frames, which reflect motion information.<br>
</p><div class="lazyYT" data-youtube-id="5VyLAH8BhF8" data-youtube-title="Lecture 06 - Optical Flow" data-width="480" data-height="270" data-parameters="feature=oembed&amp;wmode=opaque"></div>

username: Gkv
cooked: <p>So in p3d, flow modality is for getting the motion description of the video and rgb is for the spatial features?</p>

username: zhanghaoinf
cooked: <p>yes, you are right.<br>
One more thing, optical flow can reflect motion patterns to some extents, but it is different from motion descriptor like dense trajectory. There exist some implementations to extract optical flow, such as fast-flow, brox-flow and TVL1. This part I didn’t study to much. Here are some references:<br>
[1]. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks [Flow-Net v2] CVPR’17<br>
[2]. Flownet learning optical flow with convolutional networks [Flow-Net v1] ICCV’15<br>
[3]. Fast optical flow using dense inverse search [fast-flow] ECCV’16<br>
[4]. TV-L1 Optical Flow Estimation, Image Processing’13<br>
[5]. High accuracy optical flow estimation based on a theory for warping. [brox-flow] ECCV’04</p>

username: Esam
cooked: <aside class="quote no-group" data-username="zhanghaoinf" data-post="8" data-topic="15506">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhanghaoinf/48/4096_2.png" class="avatar"> zhanghaoinf:</div>
<blockquote>
<p>w</p>
</blockquote>
</aside>
<p>Hi Zhanhaoinf,</p>
<p>Can the model run in single GPU easily? or does require multiple GPUs?</p>

username: MaMaMaZige
cooked: <p>hi, I meet the same question as you, and I find a pytorch implement of C3D. The link is following:<br>
<a href="https://github.com/DavideA/c3d-pytorch" rel="noopener nofollow ugc">https://github.com/DavideA/c3d-pytorch</a></p>

username: DeepLearner007
cooked: <p>Greetings!<br>
I am interested in replicating the C3D paper by Du Tran. The original repository is in caffe. I request you to please share the fast.ai or PyTorch implementation for the same.<br>
I was able to find the following resources:<br>
———————————————<br>
PyTorch-Video-Recognition</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/jfzhang95/pytorch-video-recognition">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/jfzhang95/pytorch-video-recognition" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/b906ce55396400c94c01ddd7f9881ead29d796a0b323418790e451aa7c48f59b/jfzhang95/pytorch-video-recognition" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/jfzhang95/pytorch-video-recognition" target="_blank" rel="noopener nofollow ugc">GitHub - jfzhang95/pytorch-video-recognition: PyTorch implemented C3D, R3D,...</a></h3>

  <p>PyTorch implemented C3D, R3D, R2Plus1D models for video activity recognition. - GitHub - jfzhang95/pytorch-video-recognition: PyTorch implemented C3D, R3D, R2Plus1D models for video activity recogn...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Repository containing models lor video action recognition, including C3D, R2Plus1D, R3D, inplemented using PyTorch (0.4.0)<br>
Trained on UCF101 and HMDB51 datasets<br>
———————————————<br>
Pytorch porting of C3D network, with Sports1M weights<br>
Defining the C3D model as per the paper, not the complete implementation</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/DavideA/c3d-pytorch">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/DavideA/c3d-pytorch" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/0bf8ce85cffe9907cf62eb10a347fb6d9ade9f92782957c208418cfeff464a8c/DavideA/c3d-pytorch" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/DavideA/c3d-pytorch" target="_blank" rel="noopener nofollow ugc">GitHub - DavideA/c3d-pytorch: Pytorch porting of C3D network, with Sports1M...</a></h3>

  <p>Pytorch porting of C3D network, with Sports1M weights - GitHub - DavideA/c3d-pytorch: Pytorch porting of C3D network, with Sports1M weights</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
———————————————</p>
<p>I am aware that developments have been made in the field of action recognition since 2015, but I am specifically interested in the C3D paper by FAIR.<br>
Thanking you (all) in anticipation</p>