I can't figure out how to correctly submit data to the neural network. And correctly make the architecture of the model

welet: <p>Hello pytorch community! I have a number of questions. The answer to which I could not find. That’s why I’m here!<br>
1.Perhaps I’ll start with the goal of my neural network. I want to get a neural network. Which predicts the position of the maya controls based on the sound wave.<br>
2. What do I have available.<br>
2.1. audio files of various lengths. From 1 second to 7<br>
2.2. There are data from 35 face controls of the 3D model. Who repeat the phrase with the help of lips. transformations and fractions. The data is grouped like this. One sheet contains data on 35 controls. each control has 6 x,y,z transform and x,y,z rotate values.<br>
3. format of tenzers here the first problems begin: I can get the most different matrix of values ​​from controls. 3d. or 2d. It all depends on the two words in the XD code, now I have a 2d tensor that contains [x,4] where, x is the number of fragments that directly correlates with the duration. Here is an example from my code:<br>
dimension mfcc ep15_sc007.txt torch.Size([1431, 4])<br>
dimension txt ep15_sc007.txt torch.Size([1431, 4])<br>
dimension mfcc ep15_sc009.txt torch.Size([6148, 4])<br>
dimension txt ep15_sc009.txt torch.Size([6148, 4]).<br>
however, I can also get a 3d tensor that is more readable for humans, but I don’t know if it’s in the machine format [x, 53,4] where x is the number of animation keys in the scene. Example:<br>
dimension mfcc ep15_sc007.txt torch.Size([27, 53, 4])<br>
dimension txt ep15_sc007.txt torch.Size([27, 53, 4])<br>
dimension mfcc ep15_sc009.txt torch.Size([116, 53, 4])<br>
dimension txt ep15_sc009.txt torch.Size([116, 53, 4])<br>
4. the problem is. What I want is to get what I have in txt in order to return it to maya and get lip sync.<br>
5. and I just can’t find the right model that will do what I need because of the small amount of data this topic is in principle (maybe I didn’t find something XD)<br>
6. I will also show the code that I wrote for my model. So that more experienced people point me to the correct vector</p>
<blockquote>
<p>torch.cuda.empty_cache()<br>
gc.collect()<br>
def split_mfcc(mfcc_tensor, frame_size):<br>
frames = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
num_frames= math.ceil(mfcc_tensor.size(1) / frame_size)</p>
<p>for me in the range(num_frames):<br>
start_index = i * frame size<br>
end_index = start_index + frame size<br>
kadr = mfcc_tensor[:, start_index:end_index]<br>
frames.add(frame)</p>
<h1>Checking the frame size and adding padding to each row if necessary</h1>
<p>last_frame_size = mfcc_tensor.size(1) % frame size<br>
if last_frame_size &gt; 0:<br>
filling = torch.zeros(mfcc_tensor.size(0), frame_size - last_frame_size)<br>
filled frames = torch.cat ([frames[-1], fill], darken=1)<br>
frames[-1] = filled frame_frames</p>
<p>returned frames</p>
<p>class MyModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super (my modeler, myself).<strong>initialize</strong>()</p>
<h1>Model architecture definition</h1>
<p>self.conv1 = nn.Conv1d(in_channels=1, out_channels=53, kernel_size=3, stride=1, padding=0)<br>
self.conv2 = nn.Conv1d(53, 53, kernel_size=2, stride=1, padding=0)<br>
self.rnn = nn.GRU(53, 53, num_layers=5, batch_first=True)<br>
self.fc1 = nn.Linear(53, 25)<br>
self.fc2 = nn.Linear(25, 4)<br>
self.dropout = nn.Dropout(0.5)<br>
itself.leaky_relu = nn.LeakyReLU()</p>
<p>definition of direct(self, x):<br>
x = x.remove the lock(1)<br>
x = self.conv1(x)<br>
x = self.leak_relu(x)<br>
x = self.conv2(x)<br>
x = self.leak_relu(x)<br>
x = x.set(0, 2, 1) # Setting the boundary line by passing to recurrent values<br>
x, _ = self.rnn(x)<br>
x = x[:, -1, :] # Use only the subsequent output from recurrent words<br>
x = self.dropout(x)<br>
x = self.fc1(x)<br>
x = self.leak_relu(x)<br>
x = self.fc2(x)<br>
x = self.leak_relu(x)<br>
believe x</p>
<h1>Creating an instance of the model</h1>
<p>model = MyModel()<br>
<span class="hashtag-raw">#device</span> = torch.device(“cuda:0”, i.e. torch.cuda.is_available() instead of “cpu”)<br>
<span class="hashtag-raw">#model</span>.to (device)</p>
<h1>Definition of the loss function</h1>
<p>criterion = nn.MSELoss()</p>
<h1>Definition</h1>
<p>optimizer = torch.optim.Adam(model.parameters(), lr=0.001) <span class="hashtag-raw">#SGD</span><br>
percentage deviation = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
precision of the post = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
l2_loss_epoch = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for the epoch in the range(0, 500):<br>
sort = ‘/compressed/disk/My disk/dataset/txt_sorting/’<br>
txt_file_names = os.listdir(sort)<br>
transformation of matrices = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
wav_mapping = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
scene_list = <span class="chcklst-box fa fa-square-o fa-fw"></span> list of scenes = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for the file_name in txt_file_names:<br>
line_mapping = <span class="chcklst-box fa fa-square-o fa-fw"></span> string matching<br>
list of frames = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
with codecs.open(sort + filename, “r”, encoding=“utf-8”) as a file:<br>
for a line in a file:</p>
<p>line_mapping.add(line.band())</p>
<h1>Converting a string to a list</h1>
<p>data = ast.literal_eval(string)</p>
<p>contol_list =<span class="chcklst-box fa fa-square-o fa-fw"></span> list of messages to manage<br>
for data management:<br>
contol_list.extend(ast.literal_eval(control.get(‘4’)))<br>
contol_list.extend(ast.literal_eval(control.get(‘5’)))</p>
<p>frame_list.add(contol_list)<br>
scene_list.add(frame_list)<br>
line_count = len(line_mapping)<br>
line_tensors = torch.tensor(list of frames)</p>
<h1>Tensor dimensions</h1>
<h1>Tensor dimensions</h1>
<p>num_frames, num_controls = line_tensors.size()</p>
<h1>Dividing a tensor into 27 53x4 tensors</h1>
<p>list of frames = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
frame size = 4</p>
<p>for me in the range(num_frames):<br>
frame_tensor = torch.zeros(53, frame_size) # Creating an empty 53x4 tensor</p>
<h1>We get one line from the original tensor</h1>
<p>control_line = linear tensors[i]</p>
<h1>Splitting the string into separate time frames of size 4</h1>
<p>for me in the range(53):<br>
start_index = j * time meter<br>
end_index = start_index + frame size</p>
<p>if end_index &lt;= num_controls, then:<br>
frame_tensor[j] = control_line[start index:end index]<br>
more:</p>
<h1>If there are not enough values, fill in the paddings</h1>
<p>num_missing_values = end index - num_controls<br>
frame_tensor[j, :frame_size - num_missing_values] = control_line[start_index:]<br>
<span class="hashtag-raw">#read</span>(frame_tensor.size())<br>
frame_list.extend(frame_tensor)</p>
<p>wav_name = email name.remember(“.txt”, “.wav”)</p>
<p>audio_path = ‘/writable/disk/My disk/dataset/wav/’ + wav_name<br>
audio_samples, my_sr = librosa.load(audio_path, sr=48000)<br>
audio_samples, _ = librosa.effects.trim(audio_samples) # removing the silence<br>
duration = len(audio_samples) / my_sr<br>
frames_to_time = int(duration * 25)</p>
<h1>MFCC assignment</h1>
<p>mfcc = librosa.feature.mfcc(y=audio_samples, sr=my_sr, n_mfcc=53, n_fft=2048, hop_length=485)<br>
jump length = 2048 / 4</p>
<h1>Converting MFCC to spectrogram</h1>
<p><span class="hashtag-raw">#mel</span> = librosa.feature.inverse.mfcc_to_mel(mfcc)</p>
<h1>Restoring audio from a Stranded spectrogram</h1>
<p><span class="hashtag-raw">#audio_reconstructed</span> = librosa.feature.inverse.mel_to_audio(mel, sr=my_sr)</p>
<p>list_mfcc_frames = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
mfcc_tensor_data = torch.tensor(mfcc)<br>
frame size = 4<br>
frames = split_mfcc(mfcc_tensor_data, frame_size)</p>
<p>for a frame in frames:<br>
list_mfcc_frames.extend(frame)</p>
<p>new_tensors_mfcc = torch.stack(list_mfcc_frames)#.cuda()<br>
new_frame_list = torch.stack(list of frames)#.cuda()</p>
<h1>we add a value from new_frame_list that is added to the dimension series of the new_tensors_mfcc tensor due to the fact that the animator beat so that the animator added animation for 10+ frames.</h1>
<p>if new_tensors_mfcc.size() != new_frame_list.size():<br>
min_frames = minimum(new_tensors_mfcc.size(0), new_frame_list.size(0))<br>
new_frame_list = new_frame_list[:miniature frames]<br>
<span class="hashtag-raw">#read</span>(new_tensors_mfcc.size(), new_frame_list.size())<br>
output data = model(new_tensors_mfcc)<br>
<span class="hashtag-raw">#furnace</span>(outputs.size())<br>
page = criterion(source data, new_frame_list)</p>
<h1>Achievement of L2-regulation</h1>
<p>l2_loss = sum(torch.norm(parameter) for the parameter in model.parameters())</p>
<h1>General loss function</h1>
<p>total_loss = loss + l2_loss</p>
<h1>Back propagation and updating of weights</h1>
<p>the optimizer.zero_grad()<br>
total_loss. total loss.reverse()<br>
optimizer.step()</p>
<p>print(‘generate’,outputs.tolist())<br>
print(‘text link’,new_frame_list.tolist())<br>
l2_loss_epoch.add(l2_loss.item())</p>
<p>use = torch.eq(output data, new_frame_list)</p>
<h1>Improve accuracy by calculating the average value received from supervisors.</h1>
<p>accuracy = torch.mean(correct.float()).cpu()<br>
accuracy_epoch.add (exactly)<br>
<span class="hashtag-raw">#percentage</span> deviation.add(percentage deviation)</p>
<h1>Saving the model every epoch</h1>
<p>torch.save(model.state_dict(), ‘/compressed/disk/My disk/dataset/model/best_model_lip_sing.pt’)<br>
prints(F"Epoch: {epoch +1}, loss: {loss.element()}, 2l_loss: {total_loss.element()}, full result: {accuracy} %" , file_name) #, preliminary deviation: {minimum(percentage deviation, 100)}</p>
<h1>Audio playback</h1>
<p><a class="hashtag-cooked" href="/c/audio/9" data-type="category" data-slug="audio" data-id="9"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>audio</span></a>(audio_reconstructed, rate=my_sr)</p>
<h1>Changing the accuracy of graphs and l2_loss</h1>
<p>plt.figure(figsize=(15, 13))<br>
plt.subtitle(2, 1, 1)<br>
plt.graph(accuracy_epoch, label=‘Accuracy’)<br>
plt.xlabel(“every 60 hours 1 cycle. Each point, the number of one-eyed in the number of somewhere 0,0 is a full-scale result, and 1 is a full-scale connection”)<br>
plt.ylabel(“Point”)<br>
plt.name(“Exactly by epochs”)<br>
plt.legend()</p>
<p>plt.subtitle(2, 1, 2)<br>
plt.grachik(l2_loss_epoch, label=‘Point L2’)<br>
plt.xlabel(“EPoch”)<br>
plt.ylabel(‘Point L2’)<br>
plt.title(‘Point L2’)<br>
plt. legend()<br>
model.eval()<br>
using torch.no_grad():</p>
<p>sort = ‘/content/disk/My disk/data set/text set/sorting/txt_sorting/’<br>
txt_file_names = os.listdir(sort)<br>
transformation of matrices = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
wav_mapping = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
scene_list = <span class="chcklst-box fa fa-square-o fa-fw"></span> list of scenes = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for the file_name in txt_file_names:<br>
line_mapping = <span class="chcklst-box fa fa-square-o fa-fw"></span> string matching<br>
list of frames = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
with codecs.open(sort + filename, “r”, encoding=“utf-8”) as a file:<br>
for a line in a file:</p>
<p>line_mapping.add(line.band())</p>
<h1>Converting a string to a list</h1>
<p>data = ast.literal_eval(string)</p>
<p>contol_list =<span class="chcklst-box fa fa-square-o fa-fw"></span> list of messages to manage<br>
for data management:<br>
contol_list.extend(ast.literal_eval(control.get(‘4’)))<br>
contol_list.extend(ast.literal_eval(control.get(‘5’)))</p>
<p>frame_list.add(contol_list)<br>
scene_list.add(frame_list)<br>
line_count = len(line_mapping)<br>
line_tensors = torch.tensor(list of frames)</p>
<h1>Tensor dimensions</h1>
<h1>Tensor dimensions</h1>
<p>num_frames, num_controls = line_tensors.size()</p>
<h1>Dividing a tensor into 27 53x4 tensors</h1>
<p>list of frames = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
frame size = 4</p>
<p>for me in the range(num_frames):<br>
frame_tensor = torch.zeros(53, frame_size) # Creating an empty 53x4 tensor</p>
<h1>We get one line from the original tensor</h1>
<p>control_line = linear tensors[i]</p>
<h1>Splitting the string into separate time frames of size 4</h1>
</blockquote>