Question:
The model is overfitting very fast. With just 10 epochs my model gets 97% accuracy from train data but only 62% on test data. I have tried changing the activation, reduce and increase the learning rate but the result is even worse and it never passes 62% accuracy. Is there any technique to increase the accuracy?
import tensorflow as tf
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

Dataset=pd.read_csv('train.csv')
Dataset.dropna(inplace=True)

X = Dataset[["text"]]
y = Dataset[["target"]]

import re

def Remove_Url(string):
    return re.sub(r'(https|http)?:\/\/(\w|\.|\/|\?|\=|\&|\%|\-)*\b', '', string)

def Handle_Tags(string):
    pattern = re.compile(r'[@|#][^\s]+')
    matches = pattern.findall(string)
    tags = [match[1:] for match in matches]
    # Removing tags from main string
    string = re.sub(pattern, '', string)
    # More weightage to tag by adding them 3 times
    return string + ' ' + ' '.join(tags) + ' '+ ' '.join(tags) + ' ' + ' '.join(tags)

import demoji
demoji.download_codes()

def Handle_emoji(string):
    return demoji.replace_with_desc(string)

def Remove_html(string):
    return re.sub(r'<.*?>|&([a-z0-9]+|#[0-9]{1,6}|#x[0-9a-f]{1,6});', '', str(string))

import nltk
from nltk.corpus import stopwords
from nltk.stem.snowball import SnowballStemmer

nltk.download('punkt')
nltk.download('stopwords')
stemmer  = SnowballStemmer('english')

stopword = stopwords.words('english')

def Remove_StopAndStem(string):
    string_list = string.split()
    return ' '.join([stemmer.stem(i) for i in string_list if i not in stopword])

def Remove_UC(string):
    thestring = re.sub(r'[^a-zA-Z\s]','', string)
    thestring = re.sub(r'\b\w{1,2}\b', '', thestring)
    return re.sub(' +', ' ', thestring)

# Step 1. Remove Url
X_clearned = X['text'].apply(Remove_Url)
# Step 2. Handle Tags
X_clearned = X['text'].apply(Handle_Tags)
# Step 3. Handle emoji's
X_clearned = X['text'].apply(Handle_emoji)
# Step 4. Remove HTML Tags
X_clearned = X['text'].apply(Remove_html)
# Step 5. Remove Stopwords and Stemming
X_clearned = X['text'].apply(Remove_StopAndStem)
# Step 6. Removing Useless Characters
X_clearned = X['text'].apply(Remove_UC)

X_new_Data = X_clearned.to_frame()
X_one_hot = pd.get_dummies(X_new_Data)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_one_hot, y,  test_size=0.2,  random_state=42)

# Set random seed
tf.random.set_seed(42)

# 1. Create the model
model_4 = tf.keras.Sequential([
  tf.keras.layers.Dense(4,activation=tf.keras.activations.relu),
  tf.keras.layers.Dense(4,activation=tf.keras.activations.relu),
  tf.keras.layers.Dense(1,activation='sigmoid')
])

# 2. Compile the model
model_4.compile(loss=tf.keras.losses.BinaryCrossentropy(),
                optimizer=tf.keras.optimizers.Adam(),
                metrics=['accuracy'])

# 3. Fit the model
history_4 = model_4.fit(X_train, y_train, epochs=10, batch_size=96,validation_data=(X_test, y_test),)

Context: If you are working with text, higher accuracy probably could be reached by using a pretrained transformer model. Usually it achieves better results than simple sequential models. You can find pretrained models on TF Hub. As for examples of using them and fine-tuning to your specific task, you can look at Kaggle or Keras code examples (Natural Language Processing 2).


How to predict in grayscale image with tensorflow lite?

Question:
Hello I’m using tensorflow lite object detectipn api very well on color image.
but i want to predict with grayscale image
but having problem in predicting it with sampel code that tensorflow provid(I presume)
interactive_eager_few_shot_od_training_colab.ipynb - Colaboratory (google.com) 2
to predict, I changed grayscale like (1,720,960,1)
but having error when it goes to detect function.
How can I prdect just like color picture on gray scale image?

Context:
Hi @p_jung
Do you mean in python or Java API?
If Python then have you tried converting the image with:
image = Image.open(“path”).convert(“RGB”)
Check also here:

      pillow.readthedocs.io

Image Module 3

  The Image module provides a class with the same name which is used to represent a PIL image. The module also provides a number of factory functions, including functions to load images from files, a...

Or if you have image loaded you can do something like:
image = Image.open(""path)
tmpImg = np.zeros((image.shape[0],image.shape[1],3))
tmpImg[:,:,0] = image[:,:,0]
tmpImg[:,:,1] = image[:,:,0]
tmpImg[:,:,2] = image[:,:,0]
and use the tmpImg. If you need dimensions like [1,w,h,3] then check:

      TensorFlow

tf.expand_dims  |  TensorFlow Core v2.6.0 1

  Returns a tensor with a length 1 axis inserted at index axis.


Quick survey for Keras users: we’d like to hear from you!

Question:
If you’re a Keras user, please take a couple of minutes to answer this survey 23. It will help us prioritize our work to deliver the best possible product.
Besides multiple-choice questions, the survey includes a freeform feedback field where you can tell us about your suggestions in your own words.
Thank you for your help!

Context: The form is no longer accepting responses.  Not sure if the survey is now closed.


SLURM errors: failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error; GPU:0 unknown device

Question:
We have a SLURM batch file that fails with TF2 and Keras, and also fails when called directly on a node that has a GPU. Here is the Python script contents:
from datetime import date
import numpy as np
import matplotlib.pyplot as plt

import pandas as pd
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, SimpleRNN
from keras.optimizers import adam
from keras.layers import Dropout
from tensorflow.keras.callbacks import Callback, EarlyStopping
from sklearn.preprocessing import StandardScaler
from datetime import datetime, timedelta
from sklearn.metrics import r2_score, mean_squared_error, accuracy_score
from keras.layers.core import Dense, Dropout, Activation
from keras.layers.recurrent import LSTM
from keras.models import load_model
from keras.callbacks import EarlyStopping, ModelCheckpoint
import warnings
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = "3"
warnings.filterwarnings('ignore')
import tensorflow as tf
import logging
logging.getLogger('tesorflow').setLevel(logging.FATAL)
delay = 252
window = 60
factor = 15
K = 8.4
sbo = 1.25
sso = 1.25
sbc = 0.75
ssc = 0.5
r = 0.02
tran_cost = 0.0002
leverage = 1.0
start_val = 100
bo = 1
so = -1
X_pd=pd.read_pickle('./data/X_pd.pkl')
X = pd.DataFrame(columns=range(0, window))
Y = []
for tag in X_pd.columns[:1]:
    # i=0 ....len(X_pd.index)-window
    for i in range(0, len(X_pd.index) - window):
        X_example = X_pd.loc[i:i + window - 1][tag].values

        X= X.append(pd.Series(X_example), ignore_index=True)
        Y.append(X_pd.loc[i + window][tag])
    print('done %s stocks' % (tag))
Y=pd.DataFrame(Y)
#normalization
SS = StandardScaler()
features = SS.fit_transform(X.values)
X=features
X=pd.DataFrame(X)
#LSTM model
def trainLSTMModel(layers, neurons, d):
    model = Sequential()

    model.add(LSTM(neurons[0], input_shape=(layers[1], layers[2]), return_sequences=False,activation='relu'))
    #model.add(Dropout(d))

    #model.add(LSTM(neurons[1], input_shape=(layers[1], layers[2]), return_sequences=False))
    #model.add(Dropout(d))

    #model.add(Dense(neurons[2], kernel_initializer="uniform", activation='relu'))
    model.add(Dense(neurons[3], kernel_initializer="uniform", activation='relu'))
    optimizer=adam(learning_rate=0.001)
    #adam = Adam(decay=0.2)
    # predict up and down
    # model.compile(optimizer="adam", loss="binary_crossentropy", metrics=['accuracy'])
    model.compile(loss='mse', optimizer=optimizer)
    model.summary()
    return model
length=X.shape[0]
X=np.array(X)
Y=np.array(Y)
time_step = 60
d = 0.3
output=1
shape = [length,time_step, output] # feature, window, output
neurons = [64, 64, 32, 1]
epochs = 100
batch_size=10000
model = trainLSTMModel(shape, neurons, d)
#shape from [samples, timesteps] into [samples, timesteps, features]
n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))
gpu_no = 0
with tf.device('/gpu:' + str(gpu_no)):
#    sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True))
#    keras.backend.set_session(sess)

    print('model_manager: running tensorflow version: ' + tf.__version__)
    print('model_manager: will attempt to run on ' + '/gpu:' + str(gpu_no))
    model.fit(X, Y, epochs=epochs, verbose=2,batch_size=batch_size)

The log shows this:
Loading requirement: cuda10.1/toolkit/10.1.243
Loading cm-ml-python3deps/3.3.0
  Loading requirement: gcc5/5.5.0 python36
Loading tensorflow2-py36-cuda10.1-gcc/2.0.0
  Loading requirement: ml-pythondeps-py36-cuda10.1-gcc/3.3.0
    openblas/dynamic/0.2.20 hdf5_18/1.8.20 keras-py36-cuda10.1-gcc/2.3.1
    protobuf3-gcc/3.8.0 nccl2-cuda10.1-gcc/2.7.8
Loading openmpi/cuda/64/3.1.4
  Loading requirement: hpcx/2.4.0
2021-08-18 11:11:43.064175: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1

2021-08-18 11:18:08.026219: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-08-18 11:18:08.031771: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
2021-08-18 11:18:08.031811: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: node001
2021-08-18 11:18:08.031819: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: node001
2021-08-18 11:18:08.031921: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 460.73.1
2021-08-18 11:18:08.031958: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 460.73.1
2021-08-18 11:18:08.031966: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 460.73.1
2021-08-18 11:18:08.032266: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F
Using TensorFlow backend.
done A stocks
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm_1 (LSTM)                (None, 64)                16896
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 65
=================================================================
Total params: 16,961
Trainable params: 16,961
Non-trainable params: 0
_________________________________________________________________
model_manager: running tensorflow version: 2.0.0
model_manager: will attempt to run on /gpu:0
Traceback (most recent call last):
  File "stocks.py", line 99, in <module>
    model.fit(X, Y, epochs=epochs, verbose=2,batch_size=batch_size)
  File "/cm/shared/apps/keras-py36-cuda10.1-gcc/2.3.1/lib/python3.6/site-packages/keras/engine/training.py", line 1213, in fit
    self._make_train_function()
  File "/cm/shared/apps/keras-py36-cuda10.1-gcc/2.3.1/lib/python3.6/site-packages/keras/engine/training.py", line 316, in _make_train_function
    loss=self.total_loss)
  File "/cm/shared/apps/keras-py36-cuda10.1-gcc/2.3.1/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/cm/shared/apps/keras-py36-cuda10.1-gcc/2.3.1/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 75, in symbolic_fn_wrapper
    return func(*args, **kwargs)
  File "/cm/shared/apps/keras-py36-cuda10.1-gcc/2.3.1/lib/python3.6/site-packages/keras/optimizers.py", line 519, in get_updates
    for (i, p) in enumerate(params)]
  File "/cm/shared/apps/keras-py36-cuda10.1-gcc/2.3.1/lib/python3.6/site-packages/keras/optimizers.py", line 519, in <listcomp>
    for (i, p) in enumerate(params)]
  File "/cm/shared/apps/keras-py36-cuda10.1-gcc/2.3.1/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 963, in zeros
    v = tf.zeros(shape=shape, dtype=dtype, name=name)
  File "/cm/shared/apps/tensorflow2-py36-cuda10.1-gcc/2.0.0/lib/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py", line 2349, in zeros
    output = _constant_if_small(zero, shape, dtype, name)
  File "/cm/shared/apps/tensorflow2-py36-cuda10.1-gcc/2.0.0/lib/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py", line 2307, in _constant_if_small
    return constant(value, shape=shape, dtype=dtype, name=name)
  File "/cm/shared/apps/tensorflow2-py36-cuda10.1-gcc/2.0.0/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py", line 227, in constant
    allow_broadcast=True)
  File "/cm/shared/apps/tensorflow2-py36-cuda10.1-gcc/2.0.0/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py", line 235, in _constant_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/cm/shared/apps/tensorflow2-py36-cuda10.1-gcc/2.0.0/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py", line 96, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
RuntimeError: /job:localhost/replica:0/task:0/device:GPU:0 unknown device.

Why is the script not seeing the GPU?

Context:
Can you try to just list the visibile devices?

      TensorFlow

tf.config.get_visible_devices  |  TensorFlow Core v2.6.0 24

  Get the list of visible physical devices.


Simple benchmarking of model predication times

Question:
Hi,
I’m new to TensorFlow and have been trying to get a simple benchmark running to measure the predication latency for a model. My source code is as follows.
import time
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

tf.config.optimizer.set_jit(True) # Enable XLA

def run(model, inputs, iters):
    total_time = 0
    for i in range(iters):
        start = time.perf_counter()
        model(*inputs, training=False)
        end = time.perf_counter()
        print(1000*(end - start))
        total_time += end - start
    return (1000 * total_time) / iters

batch_size = 32
head_size = 64
num_heads = 8
model_dim = 512
max_len = 512

inputs = layers.Input(shape=(max_len, model_dim))
outputs = layers.MultiHeadAttention(num_heads=num_heads, key_dim=head_size, value_dim=head_size, use_bias=False)(inputs, inputs)

model = keras.Model(inputs=inputs, outputs=outputs)

x = tf.random.uniform((batch_size, max_len, model_dim), minval=0, maxval=1)
run(model, [x], 100)
print(run(model, [x], 100))

I observe that the execution latency is low for the first few iterations and then it rises to plateau at a different, but higher value (I’ve attached a plot of the execution time vs iterations below). I suspect this is due to memory management but I’m not sure. How do I go about performing such benchmarking correctly? My environment is as follows:
Software: TensorFlow 2.5.0, Python 3.8.10. I’m using a docker container running CUDA 11.
Hardware: GeForce GTX TitanX

Context:
Have you tried to analyze this with our profiler?

      TensorFlow

Optimize TensorFlow performance using the Profiler  |  TensorFlow Core


Question:
I’m just start coding with Tensorflow-Quantum (TFQ), it’s a really nice module. In the tutorial I found one can setup a Variational Quantum Circuit (VQC) by something like
# Build the Keras model.
model = tf.keras.Sequential([
    # The input is the data-circuit, encoded as a tf.string
    tf.keras.layers.Input(shape=(), dtype=tf.string),
    # The PQC layer returns the expected value of the readout gate, range [-1,1].
    tfq.layers.PQC(model_circuit, model_readout),
])

The PQC Layer seems to need to specify the operators(model_readout). The process of the tutorial circuit is

Data Encode → Parametrized Gates → Measurement

I want to build a quantum circuit with the process looks like

Data Encode → Parametrized Gates → Data Encode → Parametrized Gates → .etc → Measurement

which is a technique called “Data-Reuploading” by Adrián Pérez Salinas 1
However, the PQC Layers seems couldn’t do the way I want because it need to specified with operators. Is there any way to create a Parametrized Layer only without any operators(measurement). Thank you so much.

Context: @markdaoust Can you check if we we have a Quantum team member subscribed to the quantum tag?


Train a Parametrized Quantum Circuit with a loss function using the samples measurement

Question:
Hello,
While trying to implement this article Implementable Quantum Classifier for Nonlinear Data 3 i syumble upon a problem.
I cannot find a way to train my circuit using a loss function that is defined with the amplitudes of the final state and not on average measurement over every Qubits.
Here are the relevant parts of my code :
def gaussian_kernel(x,y,sigma) :
return np.exp(-np.abs(x-y)**2/2*sigma)
def loss_func(output,k):
l_outuput = list(output)
dico_res = {}
count = 0
for el in output[0] :
I_reg = el.numpy()[:N_qubit_F]
key = sum([2**i for i in I_reg])
if key not in dico_res.keys():
dico_res[key]=1
else:
dico_res[key]+=1
count += 1
loss = 0
for key in dico_res.keys():
loss += gaussian_kernel(dico_res[key],dico_res[key],sigma)
loss += gaussian_kernel(dico_res[key],0,sigma)
if k in dico_res.keys():
loss -= gaussian_kernel(dico_res[key],0,sigma)
loss += gaussian_kernel(dico_res[key],1,sigma)
loss += 1
return loss/count
optimizer = tf.keras.optimizers.Adam(1e-4)
model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(), dtype=tf.string),
tfq.layers.PQC(circuit,[cirq.Z(cirq.GridQubit(0,0))])
])
@tf.function
def train_step(circuit_input):
with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
sample_layer = tfq.layers.Sample()
output = sample_layer(circuit_input[0] + circuit, repetitions=100)
    loss = loss_func(output,k)
print(loss)

gradients = gen_tape.gradient(
    loss, symbols)
print(gradients)

optimizer.apply_gradients(
    zip(gradients, model.trainable_variables))

return loss

train_step(tfq.convert_to_tensor([circuit_input]))
The circuit in itself is a paramtetrized one.
Thanks a lot for your help.

Context: /cc @Michael_Broughton


Discrepancy code/schematic figure in Tensorflow Tutorial?

Question:
I am implementing a encoder-decoder transformer for language translation and using the TensorFlow tutorial as a guide: Transformer model for language understanding  |  Text  |  TensorFlow 3
I am confused at a specific point, notably where the MultiHead Class is created. There seems to be a discrepancy between the schematic figure: On the figure the q,k,v are first split and then run through the Dense layer. However, in the code itself, it seems they are first run through the dense layer, and only afterwards split up.
This is a significant difference, right?
Thanks

Context: hey @markdaoust , can you help here?


Question:
Hi all!
I’d like to be able to obtain the underlying computational graph consisting of the low-level tf operators of a tf2 model. Basically, for any given tf model architecture(along with training script), is there a straightforward way to modify my code to get a list of the tf operators called during training?

Context:
Does this do what you’re looking for?
import tensorflow as tf
fun = lambda x: tf.nn.softmax(x)
tf_fun = tf.function(fun)
graph = tf_fun.get_concrete_function(tf.constant([1.0])).graph
isinstance(concrete_fun.graph, tf.Graph) # True

graph.get_operations() # returns a list of graph operations

Notice the model’s forward pass will need to be a tf.function and you’ll need to know at least the input shapes ahead of time. If the model is a Keras model, here’s a somewhat hacky way to do that:
model = tf.keras.models.Sequential([
  tf.keras.layers.Dense(1, activation='softmax', input_shape=(1,))
])
model.predict(tf.constant([1.0])) # model.predict_function is only populated after 1 call to predict, you can also do the same thing with model.train_function

graph = model.predict_function.get_concrete_function(iter([tf.constant([1.0])])).graph # The concrete function takes an iterator
isinstance(graph, tf.Graph) # True

graph.get_operations()

For more info: Introduction to graphs and tf.function  |  TensorFlow Core


Keras custom models deteriorates after save and reload

Question:
Hi!
I noted that other persons encountered the same following problem, as me.
(python - Predictions become random after loading a custom saved Keras model? - Data Science Stack Exchange 1)
Description: I created a Keras model with a custom layer and I noticed that the results deteriorates significantly if I save and reload it.
To clarify, in the scenario 1 everything works fine every time. In scenario 1, I build the model then I compile and train it. But in scenario 2 (I build the model, save it untrained, reload it and then I compile and train it) the results deteriorates significantly.
I have been thinking about whether some of the parameters of the custom layers (i.e learning rate) are not correctly saved. But this shouldn’t be the case because I always recompile the model after reloading it. So at this point I do not know what can cause the problem.
I will appreciate any advise.

Context: Can you reproduce this with a dummy Colab example?


Question:
I’m reading some papers about openset classification in witch a model can predict what it knows (trained classes) and can gives an output “unknown” class when input is something it never trained on.
But it seems is still a state-of-art approach because I can’t find anything on the net about this output mode. Right now there is only “softmax” that is used for closed dataset. I’m wondering if there is some implementation or customization of softmax to add this new class (“unknown”) at test time.

Context:
I suggest to take a look at GitHub - iCGY96/awesome_OpenSetRecognition_list: A curated list of papers & resources linked to op 3
You could try to look at some reference code in the list or on the mentioned challenge.


Question:
Hello. This doubt may seem a little bit less researched, but here I go. I wanted to convert the stylegan generator to tensorflow js model. But this I understand that stylegan python (using TF2.x) has custom loops and objects in the generator object. I followed the steps mentioned below :
I right now have a keras model stylegan generator with weights loaded and the class has a call function that takes in noise and gives images. The definition is same as “StyleGAN_G” class in StyleGAN-Tensorflow2/stylegan.py at master · ialhashim/StyleGAN-Tensorflow2 (github.com) 10. The weights I loaded was using the tutorial given in the same repository StyleGAN-Tensorflow2/stylegan.ipynb at master · ialhashim/StyleGAN-Tensorflow2 (github.com) 7. So which format do I save the generator object (before converting it to tfjs model) ? Tensorflow seems to have a lot of save options
And after saving, what all flag values I have to set tfjsconvertor ?

Context: @Jason might be able to help


Question:
Is reflection padding unsupported in TPU?
Borrowing code from this blog post 3, I am trying to use a ReflectionPadding() layer inside my model and train it on TPUs.
The layer is defined like so:
from tensorflow import pad
from tensorflow.keras.layers import Layer

'''
  2D Reflection Padding
  Attributes:
    - padding: (padding_width, padding_height) tuple
'''
class ReflectionPadding2D(Layer):
    def __init__(self, padding=(1, 1), **kwargs):
        self.padding = tuple(padding)
        super(ReflectionPadding2D, self).__init__(**kwargs)

    def compute_output_shape(self, input_shape):
        return (input_shape[0], input_shape[1] + 2 * self.padding[0], input_shape[2] + 2 * self.padding[1], input_shape[3])

    def call(self, input_tensor, mask=None):
        padding_width, padding_height = self.padding
        return pad(input_tensor, [[0,0], [padding_height, padding_height], [padding_width, padding_width], [0,0] ], 'REFLECT')

Here’s the gist of the error trace:

image1574×74 14.2 KB

Is there a central repository when TPU/XLA unsupported ops are discussed/tracked?

Context:
For CPU and GPU in tf2xla we had something like this (but unmaintained)

      github.com

tensorflow/tensorflow 8

  master/tensorflow/compiler/tf2xla/g3doc

  An Open Source Machine Learning Framework for Everyone


Explain TFX Metadata Store data model definitions

Question:
GOAL

Explain the following definitions in plane simple english?
Many practical examples of what they can have (contain model, model hyper parameters, just and only data…)?
What each of them do?

ORIGINAL DOC
This is the original ML Metadata  |  TFX  |  TensorFlow 2 that I have issues with. In my description I will point back to this. “The Metadata Store uses the following data model to record and retrieve metadata from the storage backend.”:

ArtifactType: describes an artifact’s type and its properties that are stored in the metadata store. You can register these types on-the-fly with the metadata store in code, or you can load them in the store from a serialized format. Once you register a type, its definition is available throughout the lifetime of the store.
An Artifact: describes a specific instance of an ArtifactType, and its properties that are written to the metadata store.
An ExecutionType: describes a type of component or step in a workflow, and its runtime parameters.
An Execution: is a record of a component run or a step in an ML workflow and the runtime parameters. An execution can be thought of as an instance of an ExecutionType. Executions are recorded when you run an ML pipeline or step.
An Event: is a record of the relationship between artifacts and executions. When an execution happens, events record every artifact that was used by the execution, and every artifact that was produced. These records allow for lineage tracking throughout a workflow. By looking at all events, MLMD knows what executions happened and what artifacts were created as a result. MLMD can then recurse back from any artifact to all of its upstream inputs.
A ContextType: describes a type of conceptual group of artifacts and executions in a workflow, and its structural properties. For example: projects, pipeline runs, experiments, owners etc.
A Context: is an instance of a ContextType. It captures the shared information within the group. For example: project name, changelist commit id, experiment annotations etc. It has a user-defined unique name within its ContextType.
An Attribution: is a record of the relationship between artifacts and contexts.
An Association: is a record of the relationship between executions and contexts.

How I see them

(Please help me by correct my descriptions or let me know if they are right. The statements are how I perceive the descriptions and the questions are that I don’t understand and need to be answered)

Def ArtifactType (how I can draw down based on the documentations):

Contains the base data
Contains many iteration and multiple modified version of the base data
Defines the data types
Have properties
Sores data as metadasta in metadata storage ex.: database, in ram.
What is an ArtifactType?
What are all the possible things that it can store?
What are all the properties it has?

1 Artifact  (how I can draw down based on the documentations):

1 version of the modified data
1 version of the modified data’s properties
1 version of the modified data’s data types
1 version of a specific instance of an ArtifactType, and its properties that are written to the metadata store.
!! BUT THAN “List all Artifacts of a specific type. Example: all Models that have been trained.”  → So saved down model can also be Artifacts. This documentation is just terrible their ArtifactType and Artifact is pointing on each other whiteout explain any of them what it is. It doesn’t makes any sense.
What is an Artifact?
What are all the possible things that it can store?
What are all the properties it has?

ExecutionType:

What is a “component in a workflow”?
What is a “step in a workflow, and its runtime parameters.”
Because this TEXT area has NO workflow chart to point at while there is one to the previous section tfx/guide/mlmd#metadata_store and there is one after tfx/guide/mlmd#integrate_ml_metadata_into_your_ml_workflows .

Execution:

What is a record here?
What is a  component?
What is a component run?
What runtime parameters are we talking about?
What is Execution overall?
1 version of a specific instance of ExecutionType.
Executions save to metadata storage (ex.: RAM or database) you run an ML pipeline or step.

Event:

Is a record of the relationship between artifacts and executions.
to me it is not clear why is this step even necessary because event and execution sounds like they fulfill the same exact purpose.
to me it seems like execution saves down itself than why do we need an event to save it down again?
This is the only understandable statement in this definition “By looking at all events, MLMD knows what executions happened and what artifacts were created as a result.”

ContextType:

What is “conceptual group of artifacts and executions” ? Especially what is “conceptual” about them?
perfect examples this is what all the other description should be.

Context:

1 version of the ContextType.
Again what is this “conceptual group of artifacts and executions” ? Especially what is “conceptual” about them?
Again GREAT examples.

Attribution:

simple and understandable description
If all the elements ave and describes them self why is this necessary?

Association:

simple and understandable description
If all the elements ave and describes them self why is this necessary?

Previous recommendations

Just not helpful also the colab also not answers the basic definitions - /tfx/tutorials/mlmd/mlmd_tutorial

Context: Ccing @Robert_Crowe which might be able to help.


‘Profiler Report’ - like PyTorch-Lighting in TF.Keras interface

Question:
I found the profiler report 4 produced in the pytorch-lighting (pt) model training is convenient to debug training. AFAIK, tf has also a profiler which can be viewed via tensor board. In pt, it gives as follows:
Profiler Report

Action                  |  Mean duration (s)    |  Total time (s)
-----------------------------------------------------------------
on_epoch_start          |  5.993e-06            |  5.993e-06
get_train_batch         |  0.0087412            |  16.398
on_batch_start          |  5.0865e-06           |  0.0095372
model_forward           |  0.0017818            |  3.3408
model_backward          |  0.0018283            |  3.4282
on_after_backward       |  4.2862e-06           |  0.0080366
optimizer_step          |  0.0011072            |  2.0759
on_batch_end            |  4.5202e-06           |  0.0084753
on_epoch_end            |  3.919e-06            |  3.919e-06
on_train_end            |  5.449e-06            |  5.449e-06

Numbers are random of course. We can easily accomplish on_epoch_start/end, on_batch_start/end timing. But not sure about others (mostly). I don’t know yet if there is any such mechanism that exists already in tf. keras.

Context:
github.com/tensorflow/addons

      'Profiler Report' - like PyTorch-Lighting in TF.Keras interface.  2

        opened

        Mar 24, 2021

          innat

        discussion needed
        Feature Request

    **Describe the feature and the current behavior/state.**

I found the [profile…r report](https://pytorch-lightning.readthedocs.io/en/0.7.1/profiler.html) produced in the pytorch-lighting (`pt`) model training is very convenient to debug training. AFAIK, `tf` has also a profiler which can be viewed via `tensor board`. In `pt`, it has

```
Profiler Report

Action                  |  Mean duration (s)    |  Total time (s)
-----------------------------------------------------------------
on_epoch_start          |  5.993e-06            |  5.993e-06
get_train_batch         |  0.0087412            |  16.398
on_batch_start          |  5.0865e-06           |  0.0095372
model_forward           |  0.0017818            |  3.3408
model_backward          |  0.0018283            |  3.4282
on_after_backward       |  4.2862e-06           |  0.0080366
optimizer_step          |  0.0011072            |  2.0759
on_batch_end            |  4.5202e-06           |  0.0084753
on_epoch_end            |  3.919e-06            |  3.919e-06
on_train_end            |  5.449e-06            |  5.449e-06
```

Numbers are random of course. We can easily accomplish `on_epoch_start/end`, `on_batch_start/end` timing. But not sure about others (mostly). I don't know yet if there is any such mechanism exist already in `tf. keras`. I think it would be nice to have it.

**Relevant information**
- Are you willing to contribute it (yes/no): no
*If you wish to contribute, then read the requirements for new contributions in [`CONTRIBUTING.md`](https://github.com/tensorflow/addons/blob/master/CONTRIBUTING.md#requirements-for-new-contributions-to-the-repository)*
- Are you willing to maintain it going forward? (yes/no): no
- Is there a relevant academic paper? (if so, where): no
- Does the relevant academic paper exceed 50 citations? (yes/no): no
- Is there already an implementation in another framework? (if so, where):
- Was it part of tf. contrib? (if so, where): yes, https://pytorch-lightning.readthedocs.io/en/0.7.1/profiler.html

**Which API type would this fall under (layer, metric, optimizer, etc.)**
 `tfa.callbacks` I guess.

**Who will benefit from this feature?**
`tf.keras` users.

**Any other info.**


Multi-GPU doesn’t work for model(inputs) nor when computing the gradients

Question:
Hi,
When using multiple GPUs to perform inference on a model (e.g. the call method: model(inputs)) and calculate its gradients, the machine only uses one GPU, leaving the rest idle.
For example in this code snippet below:
import tensorflow as tf
import numpy as np
import os

# Make the tf-data
path_filename_records = 'your_path_to_records'
bs = 128

dataset = tf.data.TFRecordDataset(path_filename_records)
dataset = (dataset
           .map(parse_record, num_parallel_calls=tf.data.experimental.AUTOTUNE)
           .batch(bs)
           .prefetch(tf.data.experimental.AUTOTUNE)
          )

# Load model trained using MirroredStrategy
path_to_resnet = 'your_path_to_resnet'
mirrored_strategy = tf.distribute.MirroredStrategy()
with mirrored_strategy.scope():
    resnet50 = tf.keras.models.load_model(path_to_resnet)

for pre_images, true_label in dataset:
    with tf.GradientTape() as tape:
       tape.watch(pre_images)
       outputs = resnet50(pre_images)
       grads = tape.gradient(outputs, pre_images)

Only one GPU is used. You can profile the behavior of the GPUs with nvidia-smi. I don’t know if it is supposed to be like this, both the model(inputs) and tape.gradient to not have multi-GPU support. But if it is, then it’s a big problem because if you have a large dataset and need to calculate the gradients with respect to the inputs (e.g. interpretability porpuses) it might take days with one GPU.
Another thing I tried was using model.predict() but this isn’t possible with tf.GradientTape.

Context:
Here’s a working example of it. I ask to whoever has more than one GPU to try and check if the issue still remains.
Notebook: Working Example.ipynb 8
Saved Model:

HDF5


Improving multiclassification model prediction using real world dataset

Question:
Hi, I’m trying to create a multiclassification model to predict a particular bird species using a real world dataset of the following images that were submitted by the public. It consists of roughly 33,000 images very similar to the following.
Image 1:

Image 2:

Image 3:

Image 4:

Image 5:

Image 6:

Image 7:

Image 8:

Image 9:

Image 10:

Image 11:

Image 12:

Image 13:

The problem I’m having is that my model is incapable of getting an accuracy beyond ~58% on the training dataset, beyond which is starts to flatten out as shown here:

niMDg1479×632 83.1 KB

I also applied a cyclical learning rate but to no avail as shown above. So, I used transfer learning particularly InceptionV3 and MobileNet and the validation accuracy still didn’t improve beyond my own model. So, I went back to look at the data. From your experience from looking at the above images, which range from a max resolution of 4347x3260 to 512x512, do you think I need to clean the dataset by removing some of the images and cropping out parts of the images?
Thanks

Context:
B_FitzGerald:

The problem I’m having is that my model is incapable of getting an accuracy beyond ~58% on the training dataset

Is this really on the train dataset or on the test/validation dataset?


Decision Forest - Random Forests RAM issues

Question:
Dear TFDF developer(s),
I am happy to be trying out the TFDF package. I am having some RAM issues, I am not using Colab. I first tested it out using the Penguins data as suggested in Introducing TensorFlow Decision Forests — The TensorFlow Blog 4
with no issues. I also did a little research and found out the default num_trees=300 and max_depth=16, I believe from help(tfdf.keras.RandomForestModel). Then I moved up to what for me is a “middle of the road” size-wise sparse data matrix of 50k rows and 22k columns using the same loading method as in the blog and deleting data frames as I went. I am using a 32GB RAM ubuntu 18.04 instance and I track RAM using top. I stuck with the default settings and watched it evaporate until finally “Killed”. Not unexpected as the available RAM was heading to 0 and presumably this is a safeguard. I then built the same model using Sklearn, 300 trees & max_depth = None and RAM usage maxed out at about 5.4 GB. I then tried setting TFDF sorting_strategy=IN_NODE and was again watching the RAM increase to about 50% until I got a NameError: ‘IN_NODE’ is not defined. Regardless the memory using was already at ~16GB.
A six-fold increase in RAM seems out-of-bounds so I am wondering if there’s a memory issue that hasn’t been exposed yet or if perhaps I am doing something wrong. I am using the default settings and following the loading process from the blog, so I think I am doing things correctly. Please advise.
thanks,
Dan711

Context:
I obviously miss-stated when I said “I then built the same model using Sklearn, 300 trees & max_depth = None”, I should have said “I then built a similar model”… My mistake, but I think the content remains valid.
thanks again,
Dan711


Question:
Model parallelism in Keras does not seem possible as layers cannot be assigned to devices. Working with large data grids and large complex models eg using the Keras functional API,  means that we run out of memory on GPUs very quickly so an approach to model parallelism is essential particularly for the application of AI to science problems. It would be very helpful to understand what is happening here?
If you are interested here is paper that talks about these issues related to weather forecasting:-

      researchsquare.com

Data driven global weather predictions at high resolutions 4

  Research Square is a preprint platform that makes research communication faster, fairer, and more useful.

I first opened this as a bug under TensorFlow but have not had a response…

      github.com/tensorflow/tensorflow

      tf.device scope not working correctly 7

        opened

        Nov 2, 2020

          JohnTaylor2000

        stat:awaiting tensorflower
        type:support
        comp:keras
        comp:gpu
        TF 2.2

    Please make sure that this is a bug. As per our
[GitHub Policy](https://git…hub.com/tensorflow/tensorflow/blob/master/ISSUES.md),
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template

**System information**
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Y es
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ppc64le-linux
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
- TensorFlow installed from (source or binary)
- TensorFlow version (use command below):
- Python version: python3.7
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version:10.1.243
- GPU model and memory: V100 16GB

You can collect some of this information using our environment capture
[script](https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh)
You can also obtain the TensorFlow version with:
1. TF 1.0: `python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"`
2. TF 2.0: `python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"`

**Describe the current behavior**

tf.device command does not correctly assign a GPU device to tf.keras layers on node with 4 GPUs so cannot implement model parallelism. All layers appear on device GPU:0 with the exception of some IO based on output of tf.debugging.set_log_device_placement(True)

**Describe the expected behavior**
tf.keras layers are correctly assigned to a device.

**Standalone code to reproduce the issue**

import tensorflow as tf
from tensorflow import keras

tf.debugging.set_log_device_placement(True)

print("On GPU:1")
inputs = keras.Input(shape=(784,))
with tf.device("/device:GPU:1"): # Or GPU:1 for the 2nd GPU, GPU:2 for the 3rd etc.
   x = keras.layers.Dense(256, activation="relu")(inputs)
   print(x)
   assert x.device.endswith("/GPU:1")

Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.

I have a larger test problem that will run on a 4 GPU node. If you turn off the assert statement, then using nvidia-smi you can see that all memory and computational work is happening on GPU:0 and almost none is assigned to other GPUs. Happy to supply this code if needed.

**Other info / logs** Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

Tensor("dense/Identity:0", shape=(None, 256), dtype=float32)
Traceback (most recent call last):
  File "py_test.py", line 11, in <module>
    assert x.device.endswith("/GPU:1")
AssertionError

Context:
As far as I know, even if the model you wrote in the issue could work, it will encounter the curse of dimensions because of the transferring time, so you would expect a super low scaling efficiency for your simple implementation.
I would recommend you only do model parallelism unless you couldn’t even fit one reasonably small batch size. Because the model parallel could always expect smaller scaling efficiency than data parallel currently (from my knowledge).
Considering model parallelism, there is a work called mesh-tensorflow 10, you may refer to that for your purpose.  Secondly, you could also write a custom distributed strategy 12 if you wish (similar to TPUStrategy, but that’s too tedious for just one model testing and I wouldn’t recommend you to do so).


Question:
I am trying to get into Multi-class text classification I have the following code:
import pandas as pd
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
from nltk.stem import PorterStemmer
from tqdm import tqdm
from keras.models import Sequential
from keras.layers.recurrent import LSTM, GRU
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.embeddings import Embedding
from keras.layers.normalization import BatchNormalization
from keras.utils import np_utils
from keras.layers import GlobalMaxPooling1D, Conv1D, MaxPooling1D, Flatten, Bidirectional, SpatialDropout1D
from sklearn.model_selection import train_test_split
from keras.preprocessing import sequence
from keras.preprocessing.text import Tokenizer
from keras.callbacks import EarlyStopping
import matplotlib.pyplot as plt
from nltk.corpus import stopwords
STOPWORDS = set(stopwords.words('german'))

vocab_size = 2000
embedding_dim = 64
max_length = 100
trunc_type = 'post'
padding_type = 'post'
oov_tok = '<OOV>'

tickets = []
labels = []

df = pd.read_csv('tensorflow/Incidents_20210728.csv', sep=';')
df['Symptom'] = df['Symptom'].str.lower()

def getLemmText(text):
    tokens = word_tokenize(text)
    lemmatizer = WordNetLemmatizer()
    tokens = [lemmatizer.lemmatize(word) for word in tokens]
    return ' '.join(tokens)

def getStemmText(text):
    tokens = word_tokenize(text)
    ps = PorterStemmer()
    tokens = [ps.stem(word) for word in tokens]
    return ' '.join(tokens)

df['Symptom'] = list(map(getLemmText, df['Symptom']))
df['Symptom'] = list(map(getStemmText, df['Symptom']))

xtrain, xtest, ytrain, ytest = train_test_split(
    df['Symptom'], df['_qResolvedByGroup'],
    test_size=0.33,
    random_state=53)

tokenizer = Tokenizer(num_words=vocab_size, oov_token=oov_tok)
tokenizer.fit_on_texts(list(xtrain) + list(xtest))

xtrain_seq = tokenizer.texts_to_sequences(xtrain)
xtest_seq = tokenizer.texts_to_sequences(xtest)

word_index = tokenizer.word_index

xtrain_pad = sequence.pad_sequences(
    xtrain_seq, maxlen=max_length, padding=padding_type, truncating=trunc_type)
xtest_pad = sequence.pad_sequences(
    xtest_seq, maxlen=max_length, padding=padding_type, truncating=trunc_type)

label_tokenizer = Tokenizer()
label_tokenizer.fit_on_texts(list(ytrain))

train_label_seq = np.array(
    label_tokenizer.texts_to_sequences(ytrain), dtype=object)
test_label_seq = np.array(
    label_tokenizer.texts_to_sequences(ytest), dtype=object)

model = Sequential()
model.add(Embedding(len(word_index) + 1,
                    embedding_dim))
model.add(SpatialDropout1D(0.3))
model.add(Bidirectional(LSTM(embedding_dim,
          dropout=0.3, recurrent_dropout=0.3)))
model.add(Dense(embedding_dim, activation='relu'))
model.add(Dropout(0.8))
model.add(Dense(embedding_dim, activation='relu'))
model.add(Dropout(0.8))
model.add(Dense(68))
model.add(Activation('softmax'))
model.compile(loss='sparse_categorical_crossentropy',
              optimizer='adam', metrics=['accuracy'])

num_epochs = 10
history = model.fit(xtrain_pad, train_label_seq, epochs=num_epochs,
                    validation_data=(xtest_pad, test_label_seq), verbose=2)

But it throw the following error:

ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type list).

I tried to change np.array to np.hstack and searched a bit but couldn’t resolve the error.
I hope someone can help me here.

Context:
Can you please try
X=np.asarray(X).astype(np.float32) and Y=np.asarray(Y).astype(np.float32) before passing X and Y to model


How to solve tensorflow jit compile error

Question:
I have to use tf.function(jit_compile=true), but it seems contradicts with tf.py_function, sending out error message No registered ‘EagerPyFunc’ OpKernel for XLA_GPU_JIT devices compatible with node {{node EagerPyFunc}}){{node EagerPyFunc}}
import tensorflow as tf
import numpy as np

def my_numpy_func(x):  # This function must be numpy function, because it involves lots of scipy operations
  # tf.function
  return np.sinh(x)

# @tf.function(input_signature=[tf.TensorSpec(None, tf.float32)]) # ok, but I have to use jit_compile parameter
@tf.function(input_signature=[tf.TensorSpec(None, tf.float32)],jit_compile=True) # error because of jit_compile parameter
def tf_function(input):
  y = tf.py_function(my_numpy_func, [input], tf.float32)
  return y * y

a = tf_function(tf.constant(1.))
print(a)

error message
Traceback (most recent call last):
  File "test10.py", line 15, in <module>
    a = tf_function(tf.constant(1.))
  File "/home/sjtusmartboy/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/home/sjtusmartboy/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 957, in _call
    filtered_flat_args, self._concrete_stateful_fn.captured_inputs)  # pylint: disable=protected-access
  File "/home/sjtusmartboy/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1964, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/home/sjtusmartboy/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/home/sjtusmartboy/.local/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when trying to compile graph __inference_tf_function_8[_XlaMustCompile=true,config_proto="\n\007\n\003CPU\020\001\n\007\n\003GPU\020\0012\005*\0010J\0008\001\202\001\000",executor_type=""] on XLA_GPU_JIT: EagerPyFunc (No registered 'EagerPyFunc' OpKernel for XLA_GPU_JIT devices compatible with node {{node EagerPyFunc}}){{node EagerPyFunc}}
The op is created at:
File "test10.py", line 15, in <module>
  a = tf_function(tf.constant(1.))
File "test10.py", line 12, in tf_function
  y = tf.py_function(my_numpy_func, [input], tf.float32) [Op:__inference_tf_function_8]

Context:
This is expected - jit_compile=True uses XLA 2. There is no supported XLA lowering for tf.py_function and so it raises: tf.function  |  TensorFlow Core v2.6.0 1
The error is telling you XLA_GPU_JIT has no supported kernel/lowering for py_function. Is there a reason you need to use jit_compile=True?


Question:
https://tfhub.dev/vasudevgupta7/wav2vec2/1 4
can’t open the notebook

Context:
Thanks for bringing that up!
Let me take a look! I guess was something in the merge process


Custom model trigerring retracing warning

Question:
Hello!
Can someone please advise me how can I get rid of this warning received while training a custom model?
If I train several times a custom model with the same data, I receive the warning below once in a while.
It is strange also that the message says that only in 6 out of 8 tries it encountered a problem, although I use exactly the same data every time.
I use only tensorflow objects so I do not know what generates this message.
“WARNING:tensorflow:6 out of the last 8 calls to <function Model.make_predict_function..predict_function at 0x7f90184d5310> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to Better performance with tf.function  |  TensorFlow Core 10 and tf.function  |  TensorFlow Core v2.6.0 10 for  more details.”
Thank you,

Context: Are you using tf.data 7 API?


Question: I’m currently doing web development, it seems like the future is not very good. Do you think I should head towards the field of artificial intelligence?

Context:
I think in general you should pursue what you find fascinating and enjoy working on. Then, the question naturally arises whether you find AI/ML fascinating and enjoyable to work on
Therefore, I would suggest looking into it, reading some practical blogs and do some small projects.
From there, you can make a better assessment to ultimately answer your own question.


Tensorflow Transfer learning and fine-tuning Tutorial Predict Only 1 class

Question:
Hi, I try the colab tutorial of Transfer learning and fine-tuning 3 of tensorflow but when I try to check the metrics It shows that the model predicts only 1 class.
Here’s my google colab sample : Code 2

image845×563 66.3 KB

Context:
I don’t think labels is hot encoded on:
for images, labels in validation_dataset.unbatch(): # unbatch the test data and get images and labels
  y_labels.append(labels.numpy().argmax()) # append the index which has the largest value (labels are one-hot)

so that argmax makes every item in y_labels be a 0 (the first item is always the biggest one)
if you remove the argmax the confusion matrix will make more sense
by the way, you could also use the confusion matrix from TensorFlow if you want,  there’s an example here 2.


Could not find documents for tensorflow 2.5.0

Question:
I am trying to find the docs for tensorflow 2.5.0, but when I click the r2.5 on the version 4,  it directs me to the documents of tensorflow 2.6.0.

截屏2021-08-16 下午9.48.302880×1718 375 KB

I have check the url and the url on the r2.5 button is Module: tf  |  TensorFlow Core v2.6.0 1

Context:
This one: https://www.tensorflow.org/versions/r2.5/api_docs/python/tf 7
I guess it was just updated


In each epoch, is it necessary to go through every sample in the training data at least once?

Question:
Hi, new here. Question about training epoch in general. I’m trying to create a custom training loop from scratch, and have a question about epochs.
Let’s say I have a function that randomly extracts a number of batch size from training data. Something like this:
def random_batch(X, y, batch_size=32):
    idx = np.random.randint(len(X), size=batch_size)
    return X[idx], y[idx]

Let’s say I will train for 50 epochs, and within each epoch, I will use the above function on the dataset 1000 times because training set is 32,000 samples.
My question is, for each epoch, is it necessary to ensure that EVERY sample in the training data gets through the network once? Or is it okay if I just randomly select from 32 samples. In other words, do I need an additional step in the code where I drop the 32 samples from the training data after having them gone through the network, so that in future steps those samples won’t be learned again. This is to ensure every sample is learned once. Or is this not necessary?
Thanks and sorry for the newbie question!

Context:
Well, it’s not a newbie question at all. I am sharing a relevant snippets from the ML Design Patterns 2 book. See if that makes sense.

image626×852 127 KB

image636×875 147 KB


Valid reasoning to get probability? NLP classification

Question:
Hi there,
After going through the TensorFlow text classification with RNNs, I was wondering how I could get a sense of probability from a prediction output. For some background, it is a binary sentiment classification with the final layer being a Dense(1), and binary cross entropy with logits.
Because the final layer outputs logits, these cannot be directly interpreted as probabilities. As the tutorial points out “outputs >= 0 are positive”. However to get a sense of certainty of the model output,  I had an idea of obtaining something that might come close:
From the validation dataset I order all of the logits from lowest to highest. Then, find the index of the value that is closest to the prediction output. The place where this index falls (rescaled to percentages) is the percentage of certainty.
Code:
val_results = model.predict(validation_dataset)
predictions = model.predict(np.array(["This was a good movie."]))
number = min(val_results, key=lambda x:abs(x-predictions))
index = np.where(sorted(val_results) == number)
print(index[0]/len(val_results))

Is this reasoning sound?
Thanks

Context:
EDIT:
Since I am dealing with a binary classification, I could also use a sigmoid activation function in the output Dense layer, and correspondingly put from_logits=False in the binary_crossentropy loss function. This does give me an output from 0 to 1.


Accessing training data in custom callbacks

Question:
I’m working on super resolution generative adversarial network in which I’m trying to visualize data at the end of every epochs using custom callbacks
But things is that I’m kinda structs in developing custom callbacks , I don’t know how to access training data in custom callbacks
I have looked at on_train_batch_begin() method but i want the picture or image at the end of epoch
Is anyone have done that ? then please let me know
Thanks

Context:
Instead of using the entire training dataset, you can set aside a few samples that you would like to visualize over time and use them as needed. The following examples provide good references:

Image Super-Resolution using an Efficient Sub-Pixel CNN 2
DCGAN to generate face images 2


Merging two datasets with different dtypes

Question:
I’m referering to the tutorial dataset “ratings” for tensorflow recommenders
This has different dtypes (see below).
How is this created? I cannot concatenate two datasets with different dtypes.
{‘bucketized_user_age’: <tf.Tensor: shape=(), dtype=float32, numpy=45.0>, ‘movie_genres’: <tf.Tensor: shape=(1,), dtype=int64, numpy=array([7])>, ‘movie_id’: <tf.Tensor: shape=(), dtype=string, numpy=b’357’>, ‘movie_title’: <tf.Tensor: shape=(), dtype=string, numpy=b"One Flew Over the Cuckoo’s Nest (1975)">, ‘raw_user_age’: <tf.Tensor: shape=(), dtype=float32, numpy=46.0>, ‘timestamp’: <tf.Tensor: shape=(), dtype=int64, numpy=879024327>, ‘user_gender’: <tf.Tensor: shape=(), dtype=bool, numpy=True>, ‘user_id’: <tf.Tensor: shape=(), dtype=string, numpy=b’138’>, ‘user_occupation_label’: <tf.Tensor: shape=(), dtype=int64, numpy=4>, ‘user_occupation_text’: <tf.Tensor: shape=(), dtype=string, numpy=b’doctor’>, ‘user_rating’: <tf.Tensor: shape=(), dtype=float32, numpy=4.0>, ‘user_zip_code’: <tf.Tensor: shape=(), dtype=string, numpy=b’53211’>}

Context:
Got it, seems not to work for allready created datasets, but
tf.data.Dataset.from_tensor_slices(dict(dataframe))
does it.


Question:
Great updates from the TFX team at I/O. I am also a newbie in this area and after completing a few lessons from the Coursera MLOps Specialization, I got instantly hooked to TFX. Such an amazing framework! Data validator is my favorite component so far. @Robert_Crowe thank you for teaching it so beautifully.
In this video, I learned about TFX for NLP. As someone heavily into Computer Vision, a question arises:

Will there be core support for Vision inside TFX?

Context: FWIW, there are CIFAR10 example pipelines 8 in the tensorflow/tfx 5 repo.


Question:
Referring to the official Vertex AI x TFX example:

      TensorFlow

Simple TFX Pipeline for Vertex Pipelines  |  TensorFlow 13

The tutorial shows how to push the trained model to a GCS bucket as the last step. What if I wanted to also deploy it to Vertex AI with an endpoint enabled?
Any resources?

Context:
If you want to serve a custom model there Is:
https://codelabs.developers.google.com/vertex_custom_training_prediction#0 6


Question:
Hello,
i habe an autoencoder model which i want to visualize but the internals of the model architecture are not visual.

latent_dim = 64
class Autoencoder(Model):
def init(self, latent_dim):
super(Autoencoder, self).init()
self.latent_dim = latent_dim
self.encoder = tf.keras.Sequential([
layers.Flatten(),
layers.Dense(latent_dim, activation=‘relu’),
])
self.decoder = tf.keras.Sequential([
layers.Dense(784, activation=‘sigmoid’),
layers.Reshape((28, 28))
])
def call(self, x):
encoded = self.encoder(x)
decoded = self.decoder(encoded)
return decoded
autoencoder = Autoencoder(latent_dim)
plot_model(autoencoder)

Why is the model architecture not visual?
Thanks
Christoph

Context:
Check out these answers.

      stackoverflow.com

  How do I plot a Keras/Tensorflow subclassing API model? 5

  python, tensorflow, plot, keras

  answered by

    M.Innat

  on 08:45AM - 15 Sep 20 UTC

      stackoverflow.com

  How to visualize nested `tf.keras.Model (SubClassed API)` GAN mdeol with plot_model? 2

  python, tensorflow, keras

  answered by

    M.Innat

  on 04:07PM - 31 Mar 21 UTC


Question: Does anyone have any idea that how I can convert the output of the Tensorflow recommender system to tflite and use it in the web of flutter project?

Context:
Hi @Mohsen_Namazi
Have you seen the example here 4 and there 3? After the procedure have you checked the conversion with this guide 3?
Is there an error during the conversion?
Also it seems that there is a solution with Model Maker

      TensorFlow

Recommendation  |  TensorFlow Lite 1


Tensorflow.js converter is changing my input parameter sizes?

Question:
Hi! I’m trying to convert two models from TensorSpeech/TensorflowTTS into tensorflow.js, but the expected input dimensions seem to be set to a specific number instead of being allowed to vary. E.g. I expect the input shape for my mel spectrogram generator to be [-1, -1], but tensorflowjs_wizard converts it to [-1, 10], which only lets me input exactly 10 phonemes. The output of this spectrogram generator is a different size than the conversion wizard makes the vocoder model accept. Is there a setting in the wizard or in tensorflow.js I am overlooking?
TensorflowTTS: GitHub - TensorSpeech/TensorFlowTTS: TensorFlowTTS: Real-Time State-of-the-art Speech Synthesis for Tensorflow 2 (supported including English, Korean, Chinese, German and Easy to adapt for other languages) 5
I am using this notebook: Google Colaboratory 2 simply running fastspeech2.save and mb_melgan.save to export the spectrogram and vocoder models respectively, then using tensorflowjs_wizard to convert them to tensorflow.js models. The exact JS error I am getting is this (when trying to pass 14 phonemes to the spectrogram model instead of 10)
Error: The shape of dict['input_1'] provided in model.execute(dict) must be [-1,10], but was [1,14]

My code: hastebin

Context: Welcome to the forum and thanks for posting - sounds like a pretty cool project you are attempting there. Let me check in with folk and see what may be causing this.


How to add signature to wav2vec for serving

Question:
Hey I fine tuned a wav2vec using tensorflow and exported it to my drive . I want to import it back and add a signature but I can’t find out how to add input and ouputs .
Any help would be welcome .

Context: To specify a signature of the model, you can read it here: Using the SavedModel format  |  TensorFlow Core 3


Question:
I am learning tensorflow from online, and I wrote my own model but faced a problem.
What I want to try is

      drive.google.com

search_sentence_202108022304.csv 2

Google Drive file.

Above csv is the timeline with korean menu. Timeline format is monthdayhourmins. If user put the new date in the next year, I wants model to predict the menu that is in the past with closest date.
Colab link

      colab.research.google.com

Google Colaboratory 3

I came along til predict, but not sure I can print out the predicted text.
Please help!

Context: You can use np.argmax(model.predict(x), axis=-1) function to predict the desired output.


Question: what function is called when data is passed to a model ?

Context:
The call function I believe:

      TensorFlow

tf.keras.Model  |  TensorFlow Core v2.6.0 1

  Model groups layers into an object with training and inference features.

      keras.io

Keras documentation: The Model class 2


Tensorflow probability - Bayesian hierarchical model

Question:
I have a simple hierarchical model, currently developed in PyMC3 as follows:
a_b =  np.array([[3, 4, 6, 1, 5, 2, 9, 7],
       [1, 2, 8, 4, 5, 3, 9, 6],
       [2, 2, 3, 1, 5, 5, 9, 8],
       [2, 1, 8, 2, 9, 3, 8, 8],
       [2, 4, 9, 1, 4, 3, 5, 5],
       [1, 2, 9, 1, 3, 5, 5, 4]])

a_w =  np.array([
      [ 7, 6, 4, 9, 5, 8, 1, 3],
      [ 9, 8, 2, 5, 4, 5, 1, 3],
      [ 8, 8, 5, 9, 5, 5, 1, 2],
      [ 8, 9, 2, 8, 1, 8, 2, 2],
      [ 8, 6, 1, 9, 6, 7, 4, 4],
      [ 9, 8, 1, 9, 7, 5, 5, 6]])

aw_tc = np.sum(a_w, axis=1)
ab_tc = np.sum(a_b, axis=1)
dm_no, c_no = a_w.shape

with pm.Model() as BayesianBWM:
    w_star = pm.Dirichlet("w_star", a=0.01*np.ones(c_no))
    gamma_agg = pm.Gamma("beta_agg",alpha=0.01, beta=0.01)
    for i in range(dm_no):
        globals()[f"w{i}"]  =  pm.Dirichlet('w'+str(i), a=gamma_agg*w_star)
        globals()[f"aw{i}"] = pm.Multinomial('a_w'+str(i),n=aw_tc[i], p=globals()[f"w{i}"], observed=a_w[i,:])
        globals()[f"ab{i}"] = pm.Multinomial('a_b'+str(i),n=ab_tc[i], p= 1 / globals()[f"w{i}"], observed=a_b[i,:])

As you see, we have two observed matrices, a_w and a_b, and for each row of these matrices, we learn a w, and finally based on these w’s, we learn w_star.
I have difficulties implementing this in Tensorflow JointDistributionSequential. I want to develop it purely in TF probability and not using other high level libraries such as Edward2.
Any help is highly appreciated!

Context: /cc @Christopher_Suter


Question: How can I handle overfitting with mask layers? Because with mask layer, it has high chances of zeros present in input data

Context: Did you try leaving a portion of your data (10% to 15%) for validation purpose? You can define early stopping callback object (read here: tf.keras.callbacks.EarlyStopping  |  TensorFlow Core v2.5.1 1) and use it in training. Callback keeps track of the model accuracy on validation data while training it on the train data. If no progress is made for several epochs it stops training and restores the best weights. It should work regardless of the model architecture or task.


Review Checker using for E commerce platforms

Question: Hey Everyone…I am looking to make an algorithm that uses AI/ML to check the authenticity of reviews…does anyone have a code that does this and is willing to share it?

Context:
I guess the main challenge is not the code but the dataset to train this model.
does that exist?


Using a mixture distribution with hidden Markov models

Question:
Hello, I am currently trying to fit a hidden Markov model on a sequence of observations that consist of angles and speeds. The observations are taken from a dataset in which the amount of time between them is not constant. However, I want to model a sequence of states with a constant amount of time between them. In order to do this, I am trying to model “missing” observations, i.e. states for which an observation does not exist. I am trying to do this with a mixture distribution where the first element of each event indicates whether the observation is missing or not. Missing observations therefore correspond to [0, 0, 0], while non-missing observations correspond to [1, angle, speed], where angle is in [-pi, pi] and speed is in [0, infinity). However, I am getting an error I don’t understand when trying to compute the log probability of the hidden Markov model. Here is my current code:
# assume 2 states
num_states = 2

# Randomly initialize the initial state distribution as well as the transition probabilities
initial_probs = tf.Variable(scipy.special.softmax(rng.random([num_states])), name='initial_probs', dtype=tf.float32)
transition_probs = tf.Variable(scipy.special.softmax(rng.random([num_states, num_states]), axis=1), name='transition_probs', dtype=tf.float32)

# Initialize locations and concentrations of Von Mises distributions for turning angles
vm_locs = tf.Variable(np.zeros(num_states), dtype=tf.float32)
vm_cons = tf.Variable(np.zeros(num_states), dtype=tf.float32)

# Initialize shapes and rates of Gamma distributions for speed
gamma_shapes = tf.Variable(np.ones(num_states), dtype=tf.float32)
gamma_rates = tf.Variable(np.ones(num_states), dtype=tf.float32)

mixed_dists = tfd.Mixture(
    cat=tfd.Categorical(probs=[[0.5, 0.5]] * num_states),
    components=[
        tfd.Independent(tfd.Categorical(probs=[[[1.0, 0.0], [1.0, 0.0], [1.0, 0.0]]] * num_states, dtype=tf.float32), reinterpreted_batch_ndims=1),
        tfd.Blockwise([
            tfd.Categorical(probs=[[0.0, 1.0]] * num_states),
            tfd.VonMises(loc=vm_locs, concentration=vm_cons),
            tfd.Gamma(concentration=gamma_shapes, rate=gamma_rates)
        ], dtype_override=tf.float32)
    ]
)

hmm3 = tfd.HiddenMarkovModel(
    initial_distribution = tfd.Categorical(probs=initial_probs),
    transition_distribution = tfd.Categorical(probs=transition_probs),
    observation_distribution = mixed_dists,
    num_steps = 15
)

hmm3.log_prob(hmm3.sample()) # error here

The error I am getting is this:
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-638-bade53a804d5> in <module>
----> 1 hmm3.log_prob(observations[:15])
~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\distribution.py in log_prob(self, value, name, **kwargs)
   1294         values of type `self.dtype`.
   1295     """
-> 1296     return self._call_log_prob(value, name, **kwargs)
   1297
   1298   def _call_prob(self, value, name, **kwargs):

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\distribution.py in _call_log_prob(self, value, name, **kwargs)
   1276     with self._name_and_control_scope(name, value, kwargs):
   1277       if hasattr(self, '_log_prob'):
-> 1278         return self._log_prob(value, **kwargs)
   1279       if hasattr(self, '_prob'):
   1280         return tf.math.log(self._prob(value, **kwargs))

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\hidden_markov_model.py in _log_prob(self, value)
    487       # working_obs :: num_steps batch_shape 1 underlying_event_shape
    488
--> 489       observation_probs = observation_distribution.log_prob(working_obs)
    490       # observation_probs :: num_steps batch_shape num_states
    491

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\distribution.py in log_prob(self, value, name, **kwargs)
   1294         values of type `self.dtype`.
   1295     """
-> 1296     return self._call_log_prob(value, name, **kwargs)
   1297
   1298   def _call_prob(self, value, name, **kwargs):

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\distribution.py in _call_log_prob(self, value, name, **kwargs)
   1276     with self._name_and_control_scope(name, value, kwargs):
   1277       if hasattr(self, '_log_prob'):
-> 1278         return self._log_prob(value, **kwargs)
   1279       if hasattr(self, '_prob'):
   1280         return tf.math.log(self._prob(value, **kwargs))

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\mixture.py in _log_prob(self, x)
    281   def _log_prob(self, x):
    282     x = tf.convert_to_tensor(x, name='x')
--> 283     distribution_log_probs = [d.log_prob(x) for d in self.components]
    284     cat_log_probs = self._cat_probs(log_probs=True)
    285     final_log_probs = [

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\mixture.py in <listcomp>(.0)
    281   def _log_prob(self, x):
    282     x = tf.convert_to_tensor(x, name='x')
--> 283     distribution_log_probs = [d.log_prob(x) for d in self.components]
    284     cat_log_probs = self._cat_probs(log_probs=True)
    285     final_log_probs = [

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\distribution.py in log_prob(self, value, name, **kwargs)
   1294         values of type `self.dtype`.
   1295     """
-> 1296     return self._call_log_prob(value, name, **kwargs)
   1297
   1298   def _call_prob(self, value, name, **kwargs):

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\distribution.py in _call_log_prob(self, value, name, **kwargs)
   1276     with self._name_and_control_scope(name, value, kwargs):
   1277       if hasattr(self, '_log_prob'):
-> 1278         return self._log_prob(value, **kwargs)
   1279       if hasattr(self, '_prob'):
   1280         return tf.math.log(self._prob(value, **kwargs))

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\independent.py in _log_prob(self, x, **kwargs)
    283   def _log_prob(self, x, **kwargs):
    284     return self._reduce(
--> 285         self._sum_fn(), self.distribution.log_prob(x, **kwargs))
    286
    287   def _unnormalized_log_prob(self, x, **kwargs):

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\distribution.py in log_prob(self, value, name, **kwargs)
   1294         values of type `self.dtype`.
   1295     """
-> 1296     return self._call_log_prob(value, name, **kwargs)
   1297
   1298   def _call_prob(self, value, name, **kwargs):

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\distribution.py in _call_log_prob(self, value, name, **kwargs)
   1276     with self._name_and_control_scope(name, value, kwargs):
   1277       if hasattr(self, '_log_prob'):
-> 1278         return self._log_prob(value, **kwargs)
   1279       if hasattr(self, '_prob'):
   1280         return tf.math.log(self._prob(value, **kwargs))

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow_probability\python\distributions\categorical.py in _log_prob(self, k)
    293     k, logits = _broadcast_cat_event_and_params(
    294         k, logits, base_dtype=dtype_util.base_dtype(self.dtype))
--> 295     return -tf.nn.sparse_softmax_cross_entropy_with_logits(
    296         labels=k, logits=logits)
    297

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow\python\util\dispatch.py in wrapper(*args, **kwargs)
    204     """Call target, and fall back on dispatchers if there is a TypeError."""
    205     try:
--> 206       return target(*args, **kwargs)
    207     except (TypeError, ValueError):
    208       # Note: convert_to_eager_tensor currently raises a ValueError, not a

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow\python\ops\nn_ops.py in sparse_softmax_cross_entropy_with_logits_v2(labels, logits, name)
   4226       of the labels is not equal to the rank of the logits minus one.
   4227   """
-> 4228   return sparse_softmax_cross_entropy_with_logits(
   4229       labels=labels, logits=logits, name=name)
   4230

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow\python\util\dispatch.py in wrapper(*args, **kwargs)
    204     """Call target, and fall back on dispatchers if there is a TypeError."""
    205     try:
--> 206       return target(*args, **kwargs)
    207     except (TypeError, ValueError):
    208       # Note: convert_to_eager_tensor currently raises a ValueError, not a

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow\python\ops\nn_ops.py in sparse_softmax_cross_entropy_with_logits(_sentinel, labels, logits, name)
   4159       # The second output tensor contains the gradients.  We use it in
   4160       # _CrossEntropyGrad() in nn_grad but not here.
-> 4161       cost, _ = gen_nn_ops.sparse_softmax_cross_entropy_with_logits(
   4162           precise_logits, labels, name=name)
   4163       cost = array_ops.reshape(cost, labels_shape)

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py in sparse_softmax_cross_entropy_with_logits(features, labels, name)
  11247       return _result
  11248     except _core._NotOkStatusException as e:
> 11249       _ops.raise_from_not_ok_status(e, name)
  11250     except _core._FallbackException:
  11251       pass

~\anaconda3\envs\shark_research_project\lib\site-packages\tensorflow\python\framework\ops.py in raise_from_not_ok_status(e, name)
   6895   message = e.message + (" name: " + name if name is not None else "")
   6896   # pylint: disable=protected-access
-> 6897   six.raise_from(core._status_to_exception(e.code, message), None)
   6898   # pylint: enable=protected-access
   6899

~\anaconda3\envs\shark_research_project\lib\site-packages\six.py in raise_from(value, from_value)

InvalidArgumentError: Received a label value of -1 which is outside the valid range of [0, 2).  Label values: 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 1 0 0 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0 1 0 0 1 0 0 0 0 0 0 0 0 1 0 -1 1 0 -1 [Op:SparseSoftmaxCrossEntropyWithLogits]

The strangest part is that when I create the hidden Markov model with 14 steps and call hmm3.log_prob(hmm3.sample), I don’t get any errors. Please let me know if I should clarify anything or include more information. Any help would be appreciated!

Context:
/cc @Christopher_Suter
@markdaoust I think we could add a new probability TAG so that the TFP team could be subscribed.


Failed to load native TensorFlow runtime

Question:
Hi All,
I am new to TensorFlow - this is the stack trace of my error
 File "/home/megh/anaconda3/envs/bne/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bne.py", line 21, in <module>
    from encoder import *
  File "/home/megh/projects/entity-norm/BNE/encoder.py", line 2, in <module>
    import tensorflow as tf
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/megh/anaconda3/envs/bne/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

These are my system specifications
  Operating System: Ubuntu 20.04.1 LTS
            Kernel: Linux 5.8.0-36-generic
      Architecture: x86-64

My tensorflow versions are -
tensorboard               1.12.2                   pypi_0    pypi
tensorflow                1.12.0                   pypi_0    pypi
tensorflow-gpu            1.12.0                   pypi_0    pypi

I am running the codes on an NVIDIA GeForce RTX 2080 and CUDA 11.0
Am I facing this issue due to the mismatch between my TensorFlow version and CUDA version?
Any help will greatly be appreciated.
Thank you in advance,
Megh

Context:
Hi Megh,
Please take a look at tested build configurations 10 for pre built TF binaries.
Provided your system configuration supports AVX instructions sets, I suspect that you are having TF and Cuda incompatibility issue.
TF 2.4 is compatible with Cuda 11.0 as installed in your system.
Currently you see error because TF is looking for cuda 9.0 which is compatible with TF 1.12
I would encourage you to switch to TF 2.X versions since they are actively supported by TF team.
Hope this helps. Thanks!


Question: Hi, I only saw how to import a model training in python to c++, but I would like to build a model, training and predict, where I can found that information ? I am looking for but I never find nothing, thanks.

Context:
We have another topic like this at:

    Neural Network Training(back-propagation) in C++ General Discussion

    I’m working on a neural network implementation in C++.
In Python, we built the model sequentially via keras.
I want to learn C++, is there any good example or explanation?
The TensorFlow C++ manual is too hard to explain, so it’s quite difficult for me as a beginner to understand.
From what I understand, in the first python, the form of the neural network is saved as a .pb file.
Second, learn C++. An example from this second part is desperately needed.
Looking forward to advice from anyone…


Question:
I’m trying to input 13 integer numbers to a neural network and output 1 number depending on the 13 input numbers so I made a Flatten of input shape 13 and a Dense layer of 64 and output Dense layer with 1 neuron without any activation on any layer and I have 7 examples with 13 number for every example and 7 numbers for the corresponding output how to make the array of inputs and outputs
here is my code
`import numpy as np

from tensorflow.keras.layers import Input, Conv2D, Dense, Flatten, Dropout, GlobalMaxPooling2D, MaxPooling2D, BatchNormalization
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
xs = np.array([[2,9,5,0,5,0,6,2,4,0,3,0,8],
[2,9,6,0,9,0,7,2,4,0,3,6,9],
[2,9,7,0,5,0,2,1,3,0,0,4,2],
[2,7,0,1,0,3,1,2,7,0,0,7,0],
[2,7,0,0,9,2,7,3,1,0,0,0,6],
[2,6,9,0,6,1,7,2,7,0,1,5,5],
[2,4,6,0,4,2,3,2,4,0,0,4,8]], dtype=float).T
ys = np.array([[3,3,2,8,6,5,3]], dtype=float)
print('xs shape = ', xs.shape)
print('ys shape = ', ys.shape)
model = Sequential([
Flatten(input_shape=[xs.shape[0]]),
Dense(64),
Dense(64),
Dense(1)
])
model.compile (optimizer=‘sgd’, loss=‘mean_squared_error’)
early_stopping = EarlyStopping(monitor=‘val_loss’, patience=50)
model.summary()
model.fit(xs, ys, epochs = 1000, validation_split=0.2, callbacks=[early_stopping])
model.save(‘keras_model.h5’)`
and I got this error

Screenshot 2021-07-25 225756665×657 12.9 KB

Context:
First, both your input and target shapes are incorrect. From what you’re describing you have 7 examples with 13 input features, so your input shape should be (7, 13). There’s no need to transpose xs:
xs = np.array([[2,9,5,0,5,0,6,2,4,0,3,0,8],
[2,9,6,0,9,0,7,2,4,0,3,6,9],
[2,9,7,0,5,0,2,1,3,0,0,4,2],
[2,7,0,1,0,3,1,2,7,0,0,7,0].
[2,7,0,0,9,2,7,3,1,0,0,0,6],
[2,6,9,0,6,1,7,2,7,0,1,5,5],
[2,4,6,0,4,2,3,2,4,0,0,4,8]], dtype=float)

Your target shape should then be (7, 1). So ys should be transposed:
ys = np.array([[3,3,2,8,6,5,3]], dtype=float).T

Your neural network input shape is also incorrect, as you don’t need to specify batch size but instead specify input features:
Flatten(input_shape=[xs.shape[1]]),

Finally, you should add some non-linear activation functions, otherwise your neural network just reduces to a basic linear model:
model = Sequential([
  Flatten(input_shape=[xs.shape[1]]),
  Dense(64, activation='relu'),
  Dense(64, activation='relu'),
  Dense(1)
])

As a final warning, your model will severely overfit to your input data. You should try to collect more training data if possible, or use something different altogether.


Question:
import tensorflow as tf
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
Dataset=pd.read_csv(‘train.csv’)
Dataset_droped = Dataset.drop([“PassengerId”, “Name”, “Ticket”], axis=1)
Dataset_one_hot = pd.get_dummies(Dataset_droped)
X = Dataset_one_hot.drop([“Survived”], axis=1)
y = Dataset_one_hot[“Survived”]
X_train, X_test, y_train, y_test = train_test_split(X,
y,
test_size=0.2,
random_state=42)
tf.random.set_seed(42)
model_10 = tf.keras.Sequential([
tf.keras.layers.Dense(10),
tf.keras.layers.Dense(10),
tf.keras.layers.Dense(1)
])
model_10.compile(loss=tf.keras.losses.BinaryCrossentropy(),
optimizer=tf.keras.optimizers.Adam(),
metrics=[‘accuracy’])
history_10 = model_10.fit(X_train, y_train, epochs=10)
model_10.predict(X_test)

And then the prediction show nan. Is there a problem with this code?

Context:
You see nan  values for loss and predict because your Dataset contains missing values.
Therefore you may want to drop missing values or using imputing techniques to replace missing values before  using model.fit.
You can try adding Dataset.dropna(inplace=True) after reading train.csv
Dataset=pd.read_csv('/content/train.csv')
Dataset.dropna(inplace=True)

Please take a look at colab gist 5 that illustrates my findings. Thanks!
(I assumed you are trying kaggle’s titanic dataset )


Running Object Detection model with GPU using Task Library on Android

Question:
I have a trained Object Detection model using TFLite Model Maker, I can run the model on my Android app with Task Library but it is currently running on Phone’s CPU, how to let the model running on GPU with Task Library (I am running it from Java, not C++)
More over, I tried to run the model using Interpreter as the sample (examples/lite/examples/object_detection/android/lib_interpreter at master · tensorflow/examples · GitHub 1) but the result running on lib_interpreter is worse than the result running on Task Library, is it because of any additional tuning from Task Library when inferencing the model?
Hope you can support me.

Context:
Hi @Yen-Thanh_Le
At this moment there is no option to use Task library with GPU. TFLite team is working on this.
Also what do you mean by:

“the result running on lib_interpreter is worse than the result running on Task Library”

please define worse…


TFLite Model Maker, how to persist the trained model

Question:
I use Colab Pro to train my Object Detection using EfficientDet-Lite with TFLite Model Maker, I let the colab run 24h and export to TFLite model after it done all epoch, but if I want to train for more epoch I need to start again from the beginning. Is there anyway to save the trained model to Google Drive and reload again for further training?
More over, when it is training, if I click “stop” on Colab, the result gone, so what does it mean about this sentence on the Tutorial: “You can look at the validation accuracy during training and stop early to avoid overfitting.” (Object Detection with TensorFlow Lite Model Maker)
TFLite Model Maker is still lacking of document so I need your help.
Thank you,

Context:
Hi @Yen-Thanh_Le
Check this answer from the TensorFlow team:

    How to stop and resume object detector training(object detection model maker) General Discussion

    I am working on object detection with autonomous datasets . I want to train my model with 10000 train images,2000 test,2000 validation images.So, i will use object detection tensorflow lite model maker.
Project link : tensorflow.org/lite/tutorials/model_maker_object_detection
But the large dataset and batch size of 32, the training takes 50 epochs and takes 2 days(Step 3).I can’t keep my computer on for two days.I am running the project in jupyter notebook
How can i stop model training and ag…


Training and Val Losses increase after some epochs

Question:
I am trying to train a RNN Network for stock price prediction for my Master Thesis. I have additional input values (6), not just the stock prices by itself.
Using an LSTM Network with the “optimal” structure based on Hyperparameter Tuning with Keras Tuner, i observed a significant increase in the losses for training and validation in my case after 4000 Epochs.
My dataset consists of about 12 000 datapoints and i use the Adam optimizer with mean_absolute_error loss function
The Network is quite deep with the following layers:
LSTM (24 unit)
Dropout
LSTM (366 unit)
Dropout
LSTM (150 unit)
Dropout
Dense (1 unit)
I attached a graph of the loss (sorry for the german)
I would really like to understand what leads to this (for me) unexpected behaviour.

Verlustentwicklung_LSTMonly1728×924 137 KB

I reduced the learning rate of the adam optimizer to 0.0002. Which resulted in the V2 results in the graph, which for me reduced but did not solve the issue.
Does anybody have an idea why? I am running the same network and data with the RMSProp optimizer right now. I will get the results tomorrow.

Context:
The standard interpretation is that your network is starting to overtrain after 4000/8000 epochs.
Do you have a separate holdout “test” set? The current recommended practice is to have three datasets: training, validation, and test. You run training with the training and validation datasets, and stop when validation starts to fail. Then run the optimally valid model on the holdout test set. The validation and test loss and accuracy should roughly match.


Question:
Hi All,
I have structured training data as a pandas dataframe object. The data looks like below.

The label column is my target and I want to train a convolutional layer with this data. The input values are normalized between 0 to 1. The problem is with the label column. It takes an array of classes.
I am unable to feed this data to a convolution layer. Can anyone please help me with how I can use this data for training a model?
Thanks

Context:
You can use label encoder or one-hot encoder to represent your label.
Example for one-hot encoding,
indices = [0, 1, 2]
depth = 3
tf.one_hot(indices, depth)
# output: [3 x 3]# [[1., 0., 0.],#  [0., 1., 0.],#  [0., 0., 1.]]


Use same numpy code with numpy and tensorflow numpy emulation

Question:
I have code using the numpy API and the same code using the TF2 tensor API. Is there a cookbook somewhere that shows how to use either module with the same file?
I suspect it would be:
class A:
def use_numpy:
return self.np.mean([1,2,3])
class NP(A):
def init:
use python module package to load the numpy module for this object
class TF2(A):
def init:
use python module package to load the tensorflow.experimental module for this object
Is there an example somewhere in the TF documentation that shows how to do this? It would be very helpful for those of us trying to maintain simple codebases. Otherwise we have to have two versions of the same code, and two versions of the same unit tests.

Context:
Hi @Lance_N ,
There is no NumPy to vanilla TF guide but check out the tf.experimenal.numpy 1 API reference and the TF NumPy guide 1 for a similiar interface.


Why is my application unable to find the TFLite file?

Question:
I am currently working on an Android application (written in Java) using Google ML and TensorFlow. The application is centered around object detection during a live camera stream using the device camera, and uses a TensorFlow Lite model as a custom library to source file lists from. The TensorFlow Lite model specifically is called mobilenet_v1_0.75_192_quantized_1_metadata_1.tflite which I got here 2. My issue is that I ran the application to see if it would print the list size, and the application launches just fine; camera preview works, app doesn’t crash. However, when looking for the list size in the logcat, only these following errors are printed in my Logcat;
2021-07-30 12:19:33.343 15790-15790/? E/objectdetectio: Unknown bits set in runtime_flags: 0x8000
2021-07-30 12:19:33.754 15790-15828/com.example.objectdetection E/Perf: Fail to get file list com.example.objectdetection
2021-07-30 12:19:33.754 15790-15828/com.example.objectdetection E/Perf: getFolderSize() : Exception_1 = java.lang.NullPointerException: Attempt to get length of null array
2021-07-30 12:19:33.754 15790-15828/com.example.objectdetection E/Perf: Fail to get file list com.example.objectdetection
2021-07-30 12:19:33.754 15790-15828/com.example.objectdetection E/Perf: getFolderSize() : Exception_1 = java.lang.NullPointerException: Attempt to get length of null array
2021-07-30 12:19:33.992 15790-15835/com.example.objectdetection E/libc: Access denied finding property "persist.vendor.camera.privapp.list"

According to my research, some results state that that the file path or root is flawed in some capacity. However, I have the .tflite model placed within my asset folder, which leads me to wonder why exactly I’m receiving these errors and how I can work on resolving them. My Log.d("TAG", "onSuccess" + detectedObjects.size()); in my code below is supposed to print the object list size, but is clearly not. Here is my MainActivity.java;
public class MainActivity extends AppCompatActivity {

    private class YourAnalyzer implements ImageAnalysis.Analyzer {

        @Override
        @ExperimentalGetImage
        public void analyze(ImageProxy imageProxy) {
            Image mediaImage = imageProxy.getImage();
            if (mediaImage != null) {
                InputImage image =
                        InputImage.fromMediaImage(mediaImage, imageProxy.getImageInfo().getRotationDegrees());
                //Pass image to an ML Kit Vision API
                //...

                LocalModel localModel =
                        new LocalModel.Builder()
                        .setAssetFilePath("mobilenet_v1_0.75_192_quantized_1_metadata_1.tflite")
                        .build();

                CustomObjectDetectorOptions customObjectDetectorOptions =
                        new CustomObjectDetectorOptions.Builder(localModel)
                        .setDetectorMode(CustomObjectDetectorOptions.STREAM_MODE)
                        .enableClassification()
                        .setClassificationConfidenceThreshold(0.5f)
                        .setMaxPerObjectLabelCount(3)
                        .build();

                ObjectDetector objectDetector =
                        ObjectDetection.getClient(customObjectDetectorOptions);

                objectDetector.process(image)
                        .addOnSuccessListener(detectedObjects -> {
                            Log.d("TAG", "onSuccess" + detectedObjects.size());
                        })
                        .addOnFailureListener(e -> Log.e("TAG", e.getLocalizedMessage()))
                        .addOnCompleteListener(result -> imageProxy.close());
            }
        }
    }

    private ListenableFuture<ProcessCameraProvider> cameraProviderFuture;

    @Override
    protected void onCreate(@Nullable Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        cameraProviderFuture = ProcessCameraProvider.getInstance(this);

        cameraProviderFuture.addListener(() -> {
            try {
                ProcessCameraProvider cameraProvider = cameraProviderFuture.get();
                bindPreview(cameraProvider);
            } catch (ExecutionException | InterruptedException e) {

            }
        }, ContextCompat.getMainExecutor(this));
    }

    void bindPreview(@NonNull ProcessCameraProvider cameraProvider) {

        PreviewView previewView = findViewById(R.id.previewView);
        Preview preview = new Preview.Builder()
                .build();

        CameraSelector cameraSelector = new CameraSelector.Builder()
                .requireLensFacing(CameraSelector.LENS_FACING_BACK)
                .build();

        preview.setSurfaceProvider(previewView.getSurfaceProvider());

        Camera camera = cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, preview);
    }
}

Also, just for clarification, my application name is ObjectDetection and the IDE that I’m working on is Android Studio. This is how the .tflite model is placed within my asset folder; ObjectDetection > app > src > main > assets > ClassifierModel.tflite

Context:
Hi @Richard_Neiman
Have you uploaded somewhere the project to take a look?


How to link TensorFlow_Text ops into Android/iOS binaries?

Question:
I have a tensorflow-lite model that requires some TensorFlow_Text ops (CaseFoldUTF8 & RegexSplitWithOffsets).
While using python’s tflite interpreter, I can use
import tensorflow_text as tf_text

to load the necessary tensorflow_text binaries for inference.
But by deploying on Android, the monolithic AARs libraries from Maven Central (both lite and select-ops) do not include tensorflow_text ops, which gives me errors such as:
Op type not registered 'CaseFoldUTF8' in binary running on localhost. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.

Looking at the tflite’s guide from  Supported Select TensorFlow operators  |  TensorFlow Lite 2 , it is stated that, ‘On the runtime side, it is also required to link the TensorFlow Text or SentencePiece library into the final app or binary.’
The question is, how do I build AARs that includes TensorFlow-Text ops?
I have tried using using the tensorflow/lite/tools/build_arr.sh to build the AAR to my model, and it fails because of similar errors like “Op CaseFoldUTF8 not found”.
Also, while I haven’t tried on iOS binaries, do I need to specially build the iOS frameworks that include TensorFlow Text ops too? If so, how do I do that?

Context:
Hi @cpoohee
So you have added:

implementation ‘org.tensorflow:tensorflow-lite-select-tf-ops:2.5.0’

inside android project and it still throws an error?
Also can you provide a link to download the .tflite file to do an experiment?


Question:
Hi All,
I am trying to implement GMVAE, by using dataset on MNIST.
Purpose:
Extract class 0 and class 1 from MNIST and train a GMVAE to cluster the training samples into 2 distinct classes (Hence 2 distributions) and regenerate representations from the clusters, different from traditional VAE that fits all data into a single standard normal distribution.
I have a link to the colab notebook but I am not too sure if my implementation is indeed a GMVAE or still a normal VAE. Please advise.
Link : Google Colaboratory 6

Context: Hi @Henry_Miller , Welcome to the forum. The link attached with your post seems to be invalid.Kindly attach a valid link.


Tensorflow Model training slower on better CPU

Question:
I’m just starting to get into ML and I have the problem that I have significantly longer training times for my model (mnist) on a Ryzen 7 PRO 4750U than on my old system with Intel i5 8th Gen. Does anyone have an idea what this could be fundamentally?
Regards
Herbert

Context: Just because the CPU is “better” or “newer” doesn’t mean it will lead to faster training times. My guess would be your old system might have used MKL 5 which could explain the performance differences.