Question:
Hey everyone, I would like to share with you all a small project 18 that I had worked at my spare time, I called it “BERT as a service”, the goal is to build an end-to-end TFX pipeline for sentiment analysis using BERT. This project 18 is educational and is also aimed to provide a simple and easier reference for people that are looking to get familiar with MLOps using TFX and GCP (as was my case), but should not be that hard to tweak it to be more “production-ready”.
Here is an overview of what you will find:
An end-to-end ML pipeline, from data ingestion, data validation, transformation, training, and deployment.
Usage of specific components for infrastructure validation and hyperparameter tuning.
Orchestration using KubeFlow and the new Vertex AI
New version of TFX (1.0.0)
Colab version has a Tunner component that uses KerasTunner for HP search.
KubeFlow version has Infravalidator component to validate infra before blessing the model for deployment.
This was a very cool experience to get more familiar with MLOps concepts applied to the GCP ecosystem, if you have a similar project or any feedback I would love to know more.

Context:
This looks cool @DimitreOliveira !
@Robert_Crowe might have some insights on your project!


New Keras Example, Gradient Centralization for Better Training Performance

Question:
This new code walkthrough by me on Keras.io 4 talks about gradient centralization, a simple trick that can markedly speed up model convergence which is implemented in Keras in literally 10 lines of code. This can both speed up the training process and improve the final generalization performance of DNNs.
Further, this code example also shows the improvements on using Gradient Centralization while training on @Laurence_Moroney 's Horses v Humans dataset.

      keras.io

Keras documentation: Gradient Centralization for Better Training Performance 18

This was also my first time contributing to Keras examples  and many thanks to @fchollet and @Sayak_Paul for helping all along!

Context:
Interesting! That worked really well in the example.
I’m not quite getting the intuition for what this does/why it works. What’s your understanding? I think I understand why you keep the last axis, and what this would do with SGD.  But it’s less clear when applied through one of these more complex optimizers. Can you summarize your understanding of it (without using the word “Lipschitzness” ).


Question:
How do we process videos to feed to a Deep Learning model and train it? Can we borrow concepts from image and text models and combine those to train a video classification model? Yes, we can.
My latest example on keras.io 6 shows you how:

      keras.io

Keras documentation: Video Classification with a CNN-RNN Architecture 65

Context: Nice work Sayak!! Added to my to-read list!!


Question:
Karl Weinmeister 3 and I co-authored this blog post that discusses important concepts in Vertex AI [1]. It also shows you how to run a simple TensorFlow training job using Vertex AI.

      Google Cloud Blog

Streamline your ML training workflow with Vertex AI | Google Cloud Blog 20

  Many of us have used a local computing environment for machine learning (ML). For some problems, a local environment is more than enough. Plus, there's a lot of flexibility. Install Python, install JupyterLab, and go!

It’s really nice to see how well TensorFlow integrates with Google Cloud. First, there’s AI Platform. Second, there’s Vertex AI that provides simpler APIs with more flexibility. Third, there’s TensorFlow Cloud. It’s even nicer that TFX can fit into most of these workflows.
[1] Vertex AI  |  Google Cloud 2

Context:
Great post Sayak!
There’s also this great one about using #TFHub models with Vertex AI for inference:

      Google Cloud Blog

Serve a TensorFlow Hub model in Google Cloud with Vertex AI | Google Cloud Blog 6

  Make open-source TensorFlow Hub models ready for production by hosting them with Google Cloud's Vertex AI.

I’m very happy on how easy it is to create a rest API.
What do you think?


Using the FaceNet model for face recognition in Android

Question:
The FaceNet model has been widely adopted by the ML community for face recognition tasks. A number of Python packages are available by which can be used to leverage the powers of FaceNet.
We have used the FaceNet model to produce 128D embeddings for each face, captured in the live camera feed, so as perform face recognition in an Android app. This recognition follows the traditional approach of computing the Euclidean distance between the embeddings ( or by computing the cosine of the angle between them ).
The “Keras” of FaceNet is first converted to a TensorFlow Lite model ( Using TFLiteConverter API ) which is then used in the Android app. To perform face detection, we use Firebase MLKit’s FaceDetector. Here’s the GitHub project,

      github.com

shubham0204/FaceRecognition_With_FaceNet_Android 294

  Face Recognition using FaceNet and Firebase MLKit on Android.

Context:
Awesome! Thank you for taking the time to share this useful information.
I am learning how to set up and coordinate I.o.T. sensors and switches for controlling humidity, air exchange, and lighting in my fruiting chamber for gourmet mushrooms.
I intend to make this network of sensors and controllers secure so FaceNet would be perfect to use for faster access when I connect from outside the LAN.
I will definitely do more research about the FaceNet model. Thanks for the link!


Question:
I made a basic model in TensorFlow.js to show me how to make dice art.
I’ve been able to make basic logos out of dice!   If anyone is interested in making a more advanced model with the data, I’d love to collaborate.
Until then, check this out!

1_P0rAzGyIzUzR2YVykCnpcw4000×3000 2.23 MB

Here’s the blog post, clap if you like it!

      Medium – 19 May 21

[From our friends] Dicify AI 6

  Making art from science using AI in TensorFlow.js

    Reading time: 7 min read

Context:
Oh yeah!  I forgot to mention, it’s hanging up on @Jason 's wall!

1_a_Ev0uMVMlId51Liv6TzPA1280×718 164 KB

I loved this project.  I do think it could be even better!   I have lots of JavaScript code for the project, and it’s the capstone of my book.


Question:
Spatial Transformer Networks (STN) have been there since 2015 but I haven’t found an easy-to-follow example of it for #Keras.
On the other hand, Kevin Zakka’s implementation of STN 28 is by far one of the cleanest ones but it’s purely in TensorFlow 1. So, I decided to take the utility functions from his implementation and prepare an end-to-end example in #Keras out of it. You can find it here:

      github.com

sayakpaul/Spatial-Transformer-Networks-with-Keras 55

  This repository provides a Colab Notebook that shows how to use Spatial Transformer Networks inside CNNs build in Keras.

Comes with a Colab Notebook and also a TensorBoard callback that helps visualize the progressions of the transformations learned by STN during training.
Notice how the STN module is able to figure out transformations for the dataset that may be helpful to boost its performance -

              https://user-images.githubusercontent.com/22957388/115120399-e8084b80-9fca-11eb-97e1-c72228c3edc4.mov

Context:
Thanks, Sayuk! More concrete examples of how to combine the myriad techniques and tools helps the folks who come after you.
re: the tensorboard line in the notebook. Have you played with tensorboard.dev? If you changed that line then anyone could upload to a public hosted TensorBoard and they could share links to their specific runs. here’s a deep link to a demo colab showing how to upload to tensorboard.dev directly 1 if you’re interested.


A thread for collecting outstanding (unofficial) tutorials from around the web

Question:
This is a thread to collect and recognize outstanding (unofficial) TensorFlow tutorials (or blog posts that are mainly about specific usage of TensorFlow) from around the web.
I’ll start with a seed. I found Effective TensorFlow 2 14 to be a solid and concise tutorial. It has a nice flow and covers interesting gotchas and pitfalls.

Context:
I’d like to add this Tutorial from keras.io 3: Natural language image search with a Dual Encoder 8
it implements a model inspired by CLIP 5 using TensorFlow and TFHub. I


Question:
If you are into self-supervised learning, you already know that “representation collapse” is a real problem and is difficult to get around with simple methods. But not anymore!
Barlow Twins introduces a simple training objective that implicitly guarantees representation collapse does not happen.
Here’s my TensorFlow implementation:

      github.com

sayakpaul/Barlow-Twins-TF 21

  TensorFlow implementation of Barlow Twins (https://arxiv.org/abs/2103.03230).

With a ResNet20 as a trunk and a 3-layer MLP (each layer containing 2048 units) and 100 epochs of pre-training, I got 62.61% accuracy on the CIFAR10 test set. The pre-training total takes ~23 minutes on a single Tesla V100. Note that this pre-training does not make use of any labeled samples.
There’s a Colab Notebook inside. So, feel free to tweak it, experiment with it, and let me know. Happy to address any feedback.

Context: You’re a beast implementing these new research methods. Thank you for sharing


Question:
PAWS 3 introduces a way to combine a small fraction of labeled samples with unlabeled ones during the pre-training of vision models. With its simple and unique approach, it sets SOTA in semi-supervised learning that too with far less compute and parameters.
Here’s my implementation of PAWS in TensorFlow:

      github.com

sayakpaul/PAWS-TF 15

  Minimal implementation of PAWS (https://arxiv.org/abs/2104.13963) in TensorFlow.

For the benefit of the community, I have included all the major bits that have been used in order to make PAWS work. These recipes are largely applicable to train self-supervised and semi-supervised models at scale:

Multi-crop augmentation policy (helps a network systematically learn local to global mappings)
Class stratified sampling
WarmUpCosine LR schedule
Training with the LARS optimizer (with the correct hyperparameter choices)

Additionally, I have included a Colab Notebook 8 that walks through the Multi-crop augmentation method since it can seem daunting when you work it out for the first time.
The results are pretty promising. I encourage you, folks, to check it out.

Context:
Massive share, thank you Sayak
Loved the PAWS paper


Question:
Hi folks,
I wanted to share my new work with Pin-Yu Chen 11 (IBM Research) - “Vision Transformers are Robust Learners” .
For some time now, Transformers have taken the vision world by storm. In this work, we question the robustness aspects of Vision Transformers. Specifically, we investigate the question:

With the virtue of self-attention, can Vision Transformers provide improved robustness to common corruptions, perturbations, etc.? If so, why?

We build on top of existing works & investigate the robustness aspects of ViT. Through a series of six systematically designed experiments, we present analyses that provide both quantitative & qualitative indications to explain why ViTs are indeed more robust learners.

Paper: [2105.07581] Vision Transformers are Robust Learners 6

Code: https://git.io/J3VO0 4

Context:
Sayak_Paul:

[2105.07581] Vision Transformers are Robust Learners

Congratulations - amazing work on ViT research.

image1013×897 457 KB


Question:
It is a common belief that if we constrain vision models to perceive things as humans do, their performance can be improved. For example, in this work 9, Geirhos et al. showed that the vision models pre-trained on the ImageNet-1k dataset are biased toward texture whereas human beings mostly use the shape descriptor to develop a common perception. But does this belief always apply especially when it comes to improving the performance of vision models?
Know more in this post:

      keras.io

Keras documentation: Learning to Resize in Computer Vision 23

Context: Great share, thanks Sayak!


About the Special Interest Groups category

Question:
TensorFlow’s Special Interest Groups (SIGs) support community collaboration on particular project focuses.
Our SIGs include:

SIG I/O
SIG JVM
SIG MLIR
SIG TFjs
SIG Recommenders
SIG Models
SIG Build
SIG Addons
SIG TensorBoard
SIG Micro
SIG Keras
SIG Swift
SIG Rust

Context: I think we have 18 SIGs now community/sigs at master · tensorflow/community · GitHub 9


Customize Keras Loss to take Mean of the Regularizations intead of the Sum

Question:
I was writing some simple models and I did not want the Model Loss to be the sum of all L2 Regularizations. I wanted it to be the mean instead. My reason being that having 3 L2 Losses had a huge impact of regularization, taking the mean reduces that impact. In most courses as well, we can take the mean
Any idea on how to approach it in a manner that can generalize well
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
model = Sequential()
model.add(Dense(100, input_shape=(8,), kernel_regularizer=tf.keras.regularizers.L2(0.01)))
model.add(Dense(80, kernel_regularizer=tf.keras.regularizers.L2(0.01)))
model.add(Dense(30, kernel_regularizer=tf.keras.regularizers.L2(0.01)))
model.add(Dense(1))
model.compile(optimizer=‘adam’, loss=‘binary_crossentropy’, metrics=[‘accuracy’])
print(model.losses)
[<tf.Tensor: shape=(), dtype=float32, numpy=0.15066518>, <tf.Tensor: shape=(), dtype=float32, numpy=0.883246>, <tf.Tensor: shape=(), dtype=float32, numpy=0.4300898>]
I would want the loss to add (0.15066518 + 0.883246 + 0.4300898)/3 instead of (0.15066518 + 0.883246 + 0.4300898)

Context:
Do you want just to Reduction.None like in

      github.com

    keras-team/keras/blob/master/keras/losses.py#L546-L547

          >>> bce = tf.keras.losses.BinaryCrossentropy(from_logits=True,
          ...     reduction=tf.keras.losses.Reduction.NONE)

And then apply your custom operation?


Tsjs-tflite wasm does not work on codepen

Question:
I am trying to get a tfjs-tflite demo from here 1
to get it to work on codepen here 2
The goal is to cartoonise the image on the top row and generate its corresponding cartoon via GAN on the second row.
For some reason, it does not seem to produce a result  However when I do yarn it works fine.
Any clue as to why this is would be greatly appreciated (I am a JS newbie).

Context: I guess @Jason might be able to help here


Mlir-hlo repo cmake setup for dependent projects

Question:
This is a question related to projects that would like to depend on mlir-hlo and set it up as a submodule (while using a cmake setup).
https://github.com/tensorflow/mlir-hlo
As far as I could tell, mlir-hlo doesn’t include something equivalent to MLIRHLOConfig.cmake.in that would allow a user project to find it and see all its targets. As a result, one approach would be to set it up with proper target_link options to point to the submodule’s build dir. This creates issues with transitive dependencies and rebuilding after updates. I haven’t tried the approaches of cmake external projects ExternalProject — CMake 3.22.1 Documentation or simply an add_subdirectory.
Is there a recommended setup for projects that would like to depend on mlir-hlo via a submodule? Would a contribution of Config.cmake be acceptable in the repo or does one already exist somewhere? Thanks!

Context: I don’t know about the solution here, but we’d take a patch


TensorFlow “TypeError: Target data is missing” though dataset with 2 dimension tuple was supplied

Question:
I’m trying to use a generator- based data set:
def gen():
    return zip(samples,feature)
ds = tf.data.Dataset.from_generator(gen,output_types=tf.dtypes.float32)

model.fit(ds,
          epochs=150,
          #callbacks=[tensorboard_callback]
          )
model.save("/sda/anyone/imagenet-in-np/transformer")

whereas feature is  numpy.ndarray (2D array)
whereas feature is  numpy.ndarray (4D array)
And I get the following error:
TypeError: Target data is missing. Your model has `loss`: BinaryCrossentropy, and therefore expects target data to be passed in `fit()`.

which is strange, as the target data is actually present.
Whenever I separate the dataset to two
def gen():
    return samples
ds = tf.data.Dataset.from_generator(gen,output_types=tf.dtypes.float32)

def gen2():
    return feature
ds2= tf.data.Dataset.from_generator(gen2,output_types=tf.dtypes.float32)

model.fit(ds,ds2,
          epochs=150,
          #callbacks=[tensorboard_callback]
          )
model.save("/sda/anyone/imagenet-in-np/transformer")

I get:
    raise ValueError("`y` argument is not supported when using "
ValueError: `y` argument is not supported when using dataset as input.

Which means that TF doesn’t accept this split.
I tried
def gen():
    for element in zip(samples,feature):
        yield element
ds = tf.data.Dataset.from_generator(gen(),output_types=tf.dtypes.float32)

I get
TypeError: generator must be a Python callable.
So I tried to swap it to :
def gen():
    for element in zip(samples,feature):
        yield element
ds = tf.data.Dataset.from_generator(gen,output_types=tf.dtypes.float32)

I get again:
    TypeError: Target data is missing. Your model has `loss`: BinaryCrossentropy, and therefore expects target data to be passed in `fit()`.
python-BaseException

So how should I use the generator API?

Context:
I actually got the same error and my mistake was that the model was expecting the input and labels in a different format while I was passing them in an incorrect format.
Additionally can you provide a Minimum working example (MWE)? It would be easier to find out the problem that way.


Tensorflow transforms passes hardcoded on “main” function

Question:
Hello,
I notice at least about half a dozen tensorflow transforms passes (like tensor-list-ops-decomposition) hardcoded to work only on a single function named “main”.
(tensor_list_ops_decomposition.cc on github)
  void TensorListOpsDecompositionPass::runOnOperation() {
    auto module = getOperation();
    auto main = module.lookupSymbol<FuncOp>("main");
    if (!main) return;
    if (failed(DecomposeTensorListOps(&main.front(), module))) {
      signalPassFailure();
    }
  }

Is there an assumption that the canonical form is one where the “entry function” is named “main”? This isn’t true for an import/translation from a tf.function where the entry function has the tf.function’s name with a suffix/prefix. Should this check instead be for a function with the attribute “tf.entry_function” and should this be patched like this or better with a common utility to update all passes with such checks?
-  auto main = module.lookupSymbol<FuncOp>("main");
-  if (!main) return;
-  if (failed(DecomposeTensorListOps(&main.front(), module))) {
-    signalPassFailure();
+  for (auto func_op : module.getOps<FuncOp>()) {
+    // Just run on the entry function.
+    if (!func_op->getAttr("tf.entry_function") && func_op.getName() != "main")
+      continue;
+    if (failed(DecomposeTensorListOps(&func_op.front(), module))) {
+      signalPassFailure();
+    }
+    break;
   }

Related to this are also several instances of “main” and “tf.entry_function” hardcoded in “transforms/” and “translate/”.

Context:
We likely should provide a helper for this instead of a raw loop.
Also it isn’t clear to me what a public function (from an MLIR symbol visibility point of view) that isn’t an entry function would mean? And if so why not just filter on public ones?


Creating float tensors from BufferedImage in Java/Kotlin

Question:
Continuing a thread started on Gitter:

Hello, I want to run a Tensorflow model I found with a Java app, but I am having difficulty with getting the input just right. Below you can see the result from the layer analysis. I found a few examples for one-dimensional input (mnist) and I got another model working that required integers, but creating Tensor with dimensions {batch, height, width, channels} is a difficult task. I would like some help. The input is just a JPG, basically BufferedImage as I want to keep my options open.

Often TF Java users are looking for a snippet showing how this can be done easily, I’m sharing one here written in Kotlin (warning, I did not test it out after modifying it, but basically the logic should be good):
   fun preprocess(sourceImages: List<BufferedImage>, imageHeight: Int, imageWidth: Int, imageChannels: Int): TFloat32 {
        val imageShape = Shape.of(sourceImages.size.toLong(), imageHeight.toLong(), imageWidth.toLong(), imageChannels.toLong())

        return TFloat32.tensorOf(imageShape) { tensor ->

            // Copy all images to the tensor
            sourceImages.forEachIndexed { imageIdx, sourceImage ->

                // Scale the image to required dimensions if needed
                val image = if (sourceImage.width != imageWidth || sourceImage.height != imageHeight) {
                    val scaledImage = BufferedImage(imageWidth, imageHeight, BufferedImage.TYPE_3BYTE_BGR)
                    scaledImage.createGraphics().apply {
                        setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_NEAREST_NEIGHBOR)
                        drawImage(sourceImage, 0, 0, imageWidth, imageHeight, null)
                        dispose()
                    }
                    scaledImage
                } else {
                    sourceImage
                }

                // Converts the image to floats and normalize by subtracting mean values
                var i = 0
                for (h in 0L until imageHeight) {
                   for (w in 0L until imageWidth)  {
                       // "caffe"-style normalization
                       tensor.setFloat(image.data.dataBuffer.getElemFloat(i++) - 103.939f, imageIdx.toLong(), h, w, 0)
                       tensor.setFloat(image.data.dataBuffer.getElemFloat(i++) - 116.779f, imageIdx.toLong(), h, w, 1)
                       tensor.setFloat(image.data.dataBuffer.getElemFloat(i++) - 123.68f, imageIdx.toLong(), h, w, 2)
                   }
                }
            }
        }
    }

So the idea is simply to resample your image if it is not already of the right size and to normalize its pixel values when feeding the tensor. The “caffe”-style normalization is the one used by default by Keras in Python so the mean values to subtract were picked from Keras sources 4 directly.
UPDATED : here’s the Java version
    TFloat32 preprocess(List<BufferedImage> sourceImages, int imageHeight, int imageWidth, int imageChannels) {
        Shape imageShape = Shape.of(sourceImages.size(), imageHeight, imageWidth, imageChannels);

        return TFloat32.tensorOf(imageShape, tensor -> {
            // Copy all images to the tensor
            int imageIdx = 0;
            for (BufferedImage sourceImage : sourceImages) {
                // Scale the image to required dimensions if needed
                BufferedImage image;
                if (sourceImage.getWidth() != imageWidth || sourceImage.getHeight() != imageHeight) {
                    image = new BufferedImage(imageWidth, imageHeight, BufferedImage.TYPE_3BYTE_BGR);
                    Graphics2D graphics = image.createGraphics();
                    graphics.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_NEAREST_NEIGHBOR);
                    graphics.drawImage(sourceImage, 0, 0, imageWidth, imageHeight, null);
                    graphics.dispose();
                } else {
                    image = sourceImage;
                }

                // Converts the image to floats and normalize by subtracting mean values
                int i = 0;
                for (long h = 0; h < imageHeight; ++h) {
                    for (long w = 0; w < imageWidth; ++w)  {
                        // "caffe"-style normalization
                        tensor.setFloat(image.getData().getDataBuffer().getElemFloat(i++) - 103.939f, imageIdx, h, w, 0);
                        tensor.setFloat(image.getData().getDataBuffer().getElemFloat(i++) - 116.779f, imageIdx, h, w, 1);
                        tensor.setFloat(image.getData().getDataBuffer().getElemFloat(i++) - 123.68f, imageIdx, h, w, 2);
                    }
                }
                ++imageIdx;
            }
        });
    }

Context: Sorry I can’t add links but there’s also some example java code in the tensorflow-java models github repository. You need to drill down to the cnn FasterRcnnInception directory


Question:
I am going to build my project and data is fetched from my database with specific Project_id. and then train my model using LSTM. Epochs are clearly running but after that, It shows an Internal Server Error

admin.py
 def build(self, request, queryset):
        count = 0

        for p in queryset:
            if build_id(p.project_management.id):
                count += 1
            else:
                messages.warning(request, f"Could not build model for {p}")

        messages.success(
            request, f"Successfully built models for {count} projects")

    build.short_description = "Build models for selected Projects"

bild.py
here the model is built via a specific Project_id. Model store only model.pkl data but not completed. And other files scalar_in and scalar_out do not save in a specific folder.

def build_id(project_id):
    # get directory path to store models in
    path = fetch_model_path(project_id, True)

    # train model
    model, scaler_in, scaler_out = train_project_models(project_id)

    # ensure model was trained
    if model is None:
        return False

    # store models
    store_model(f'{path}/model.pkl', model)
    store_model(f'{path}/scaler_in.pkl', scaler_in)
    store_model(f'{path}/scaler_out.pkl', scaler_out)

    # clear current loaded model from memory
    keras_clear()

    return True

utils.py
    with open(path, 'wb') as f:
        model_file = File(f)
        pickle.dump(model, model_file)

when I Comment on the pickle.dump(model,model_file)  then model.pkl, scalar_in.pkl, and scalar_out.pkl save files with 0 kb data. If pkl files exist already with data then it removes and builds the project successfully. I debug this code and the Django debuger_tool shows that the page is temporarily moved.

output

Epoch 1/4
11/11 [==============================] - 9s 302ms/step - loss: 0.4594 - val_loss: 0.2777
Epoch 2/4
11/11 [==============================] - 2s 177ms/step - loss: 0.1039 - val_loss: 0.0395
Epoch 3/4
11/11 [==============================] - 2s 170ms/step - loss: 0.0545 - val_loss: 0.0361
Epoch 4/4
11/11 [==============================] - 2s 169ms/step - loss: 0.0414 - val_loss: 0.0551
Internal Server Error: /turboai/turboAI/jaaiparameters/
Traceback (most recent call last):
  File "E:\.Space\project\venv\lib\site-packages\django\core\handlers\exception.py", line 47, in inner
    response = get_response(request)
  File "E:\.Space\project\venv\lib\site-packages\django\core\handlers\base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "E:\.Space\project\venv\lib\site-packages\django\contrib\admin\options.py", line 616, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "E:\.Space\project\venv\lib\site-packages\django\utils\decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "E:\.Space\project\venv\lib\site-packages\django\views\decorators\cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "E:\.Space\project\venv\lib\site-packages\django\contrib\admin\sites.py", line 232, in inner
    return view(request, *args, **kwargs)
  File "E:\.Space\project\venv\lib\site-packages\django\utils\decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "E:\.Space\project\venv\lib\site-packages\django\utils\decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "E:\.Space\project\venv\lib\site-packages\django\contrib\admin\options.py", line 1723, in changelist_view
    response = self.response_action(request, queryset=cl.get_queryset(request))
  File "E:\.Space\project\venv\lib\site-packages\django\contrib\admin\options.py", line 1408, in response_action
    response = func(self, request, queryset)
  File "E:\.Space\project\TurboAnchor\turboAI\admin.py", line 125, in build
    if build_id(p.project_management.id):
  File "E:\.Space\project\TurboAnchor\turboAI\build.py", line 48, in build_id
    store_model(f'{path}/model.pkl', model)
  File "E:\.Space\project\TurboAnchor\turboAI\utils.py", line 154, in store_model
    pickle.dump(model, model_file)
TypeError: can't pickle weakref objects
[29/Oct/2021 17:50:31] "POST /turboai/turboAI/jaaiparameters/ HTTP/1.1" 500 126722

Context:
Please look at:

      github.com/tensorflow/tensorflow

      Keras model pickle-able but tf.keras model not pickle-able 85

        opened

        Nov 29, 2019

          closed

        Oct 4, 2021

          Edwin-Koh1

        stat:awaiting response
        stat:awaiting tensorflower
        type:bug
        stalled
        comp:keras
        TF 2.5

    **System information**
- Windows 10
- Tensorflow 2.0 (CPU)
- joblib 0.14.0
…- Python 3.7.5
- Keras 2.3.1

Hello everybody! This is my first post so please forgive me if I have missed something. So I'm trying to use a genetic algorithm to train and evaluate multiple NN architectures so I need to parallelize them on a multi-core CPU. Therefore I have used joblib to try to parallelize this. However, I was stuck on my tf.keras code because it wasn't pickleable. After many hours of debugging I finally realised that the tf.keras models are not pickleable whereas keras models are.

**Describe the current behavior**
The code below works but if you replaced keras with tf.keras, there will be an error:
**Could not pickle the task to send it to the workers.**

**Describe the expected behavior**
Moving forward, tf.keras should be replacing keras and therefore tf.keras should also be pickleable.

**Code to reproduce the issue**
```
#The following is a simple code to illustrate the problem:
from joblib import Parallel, delayed
import keras
import tensorflow as tf

def test():
    model = keras.models.Sequential()
    return

Parallel(n_jobs=8)(delayed(test)(i) for i in range(10)) #this works as intended

def test_tf():
    model = tf.keras.models.Sequential()
    return

Parallel(n_jobs=8)(delayed(test_tf)(i) for i in range(10)) #this will spit out the error above
```

**Other comments**
I guess a quick fix would just be to replace all the existing code with tf.keras to just keras but seeing as keras support will be discontinued and absorbed by Tensorflow 2.0, I think this should be fixed.

I suggest to test this with TF 2.6.x or TF 2.7rc


How to save a model with a custom layer?

Question:
Hi everyone,
I am facing a problem while trying to save my model that has a custom layer, I followed the same method the Francois Chollet book is using, but I got this error:
ValueError: Unable to create a dataset (name already exists).
can anyone help, please?

Context: Hi, can you share the chapter from the Deep Learning with Python (v1 or v2) book you’re referring to, as well as some code? Are you saving the model with Keras ModelCheckpoint callbacks? I’m sure we’ll be able to help.


Question: I did a quick look at the tools folder and I am interested in maintaining this folder since I have some experience with docker . My github nick name is @vulkomilev .

Context: You can start to creare and review PRs related to that folder. At some point we will add you are codeowners.


Deformable convolution and other custom ops

Question:
Recently we had a refresh over a Deformable convloution WIP PR 51 in Addons.
I’ve cherry-picked this as an example as this requires us to maintain almost 3k lines of new code in the repository.
This is maintainer-ship overhead is also quite similar to what we have with other custom kernels PRs.
As Addons is one of the few Ecosystem repositories to support custom (c++) ops and the related CI infra it is quite normal that we have this kind of proposed PRs.
But as the codeownership of these components it is generally not so stable over time we would like to not merge, as possible, these custom ops PRs also to achieve a more broad hardware coverage.
What are the alternatives? How we could collaborate when a compositional implementation has huge performance gaps?
Often this kind of issues are shared across the “extend” ecosystem like  e.g. for the EmbeddingBag:

      github.com/pytorch/xla

      lowering embeddingbag to XLA 6

        opened

        Aug 5, 2020

          shz0116

        nostale
        op lowering

The embeddingbag operation has not been lowered to XLA. I saw aten:embeddingba…g from the profiling.

      github.com/tensorflow/addons

      EmbeddingBag and Product-Key Memory Layers 1

        opened

        Oct 14, 2020

          Rocketknight1

        Feature Request
        layers

    **Describe the feature and the current behavior/state.**
FAIR have a cool paper… where they introduce [Product-Key Memory Layers](https://arxiv.org/abs/1907.05242) - these are layers that can add a huge number of parameters (100M-1B) to a network with a very minimal compute overhead.

Unfortunately, implementing them efficiently depends on the [EmbeddingBag layer](https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html) from Pytorch. This layer basically does a gather op followed by a weighted sum across the final dimension of the gather indices.

It is trivial to implement this op as a composition of two or three ops in Tensorflow, but doing so requires you to materialize the output of the gather, which in the case of Product-Key Memory layers is enormous, and usually blows out my GPU RAM. By combining these ops into a single efficient call, EmbeddingBag avoids ever materializing the extremely large pre-sum gather output. There's no efficient way to do the same in Tensorflow without a custom op.

I've already gotten a CUDA and (single-threaded) CPU implementation of EmbeddingBag working locally using the custom-op repo and associated docker image. I've verified correctness by comparing outputs and gradients to those from the manual composition of ops, and speed and memory usage are vastly improved. I could also contribute a TF implementation of the Product-Key Memory layer itself if desired.

**Relevant information**
- Are you willing to contribute it (yes/no): yes
- Are you willing to maintain it going forward? (yes/no): yes
- Is there a relevant academic paper? (if so, where):  https://arxiv.org/abs/1907.05242
- Is there already an implementation in another framework? (if so, where): Yes, EmbeddingBag is already a PyTorch layer
- Was it part of tf.contrib? (if so, where):

**Which API type would this fall under (layer, metric, optimizer, etc.)**
Layer

**Who will benefit with this feature?**
People who want to squeeze loads of parameters into their model while maintaining fast throughput and aren't worried about overfitting. The paper used it for big autoregressive NLP Transformers, but I suspect you could deploy it in a lot of other places too.

**Any other info.**
I have only implemented the portions of EmbeddingBag necessary for Product-Key Memory layers.

EmbeddingBag op and layer by Rocketknight1 · Pull Request #2352 · tensorflow/addons · GitHub (1k lines)

      github.com/tensorflow/tensorflow

      embedding_lookup cause ran out of memory  1

        opened

        Oct 5, 2020

          shz0116

        TF 2.3
        comp:tpus
        comp:xla
        stat:awaiting tensorflower
        type:bug

    I am running the following code to test embedding_lookup.
```python

# comman…d:
# python3 -m pdb embtest.py --features=1000 --nnz=30 --batch=128
#
# error:
# *** tensorflow.python.framework.errors_impl.ResourceExhaustedError:
#     Ran out of memory in memory space vmem. It should not be possible to run out of vmem - please file a bug against XLA.
#
import tensorflow as tf
import numpy as np
import sys
import os
import time

def measure(params, sp_ids, steps, thr):
  res = tf.nn.embedding_lookup([params[0:thr],params[thr:]], sp_ids, None, name="TEST1")
  print("Finished test")
  return res

if __name__ == "__main__":

  import sys
  import argparse

  parser = argparse.ArgumentParser(
     description="Measure the performance of tensorflow embeddingbag using tf.nn.embedding" )
  parser.add_argument("--features", type=int, default=10)
  parser.add_argument("--em", type=int, default=2)
  parser.add_argument("--nnz", type=int, default=2)
  parser.add_argument("--batch", type=int, default=4)
  parser.add_argument("--steps", type=int, default=1)
  parser.add_argument("--warmups", type=int, default=0)

  args = parser.parse_args()

  features     = args.features
  em           = args.em
  nnz          = args.nnz
  batch        = args.batch
  steps        = args.steps
  warmups      = args.warmups

  sp_ids = np.random.randint(0, features, (batch * nnz,))
  res = tf.zeros([batch, em])

  resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu="grpc://"+os.environ["TPU_IP"])
  tf.config.experimental_connect_to_cluster(resolver)
  tf.tpu.experimental.initialize_tpu_system(resolver)
  print("   ")
  tpus = tf.config.list_logical_devices('TPU')
  print("There are {} tpu logical devices".format(len(tpus)))
  print(tpus[0])

  with tf.device('TPU:0'):
    params = tf.random.uniform([features, em])
    res = measure(params, sp_ids, tf.constant(steps), features//2)

  print(res)
```
But got the following error:

```bash
hongzhang@shan-tf1:~$ python  embtest.py --features=1000 --nnz=30 --batch=128
Eager execution :  True
2020-10-05 08:23:42.244623: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-05 08:23:42.250601: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
2020-10-05 08:23:42.251595: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4c1dde0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-05 08:23:42.251631: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-05 08:23:42.263068: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> 10.178.175.58:8470}
2020-10-05 08:23:42.263113: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job localhost -> {0 -> localhost:38651}
2020-10-05 08:23:42.279709: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> 10.178.175.58:8470}
2020-10-05 08:23:42.279743: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job localhost -> {0 -> localhost:38651}
2020-10-05 08:23:42.280176: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:405] Started server with target: grpc://localhost:38651

There are 8 tpu logical devices
LogicalDevice(name='/job:worker/replica:0/task:0/device:TPU:7', device_type='TPU')
Traceback (most recent call last):
  File "embtest.py", line 84, in <module>
    t1 = measure(params, sp_ids, tf.constant(steps), features//2)
  File "embtest.py", line 15, in measure
    res = tf.nn.embedding_lookup([params[0:thr],params[thr:]], sp_ids, None, name="TEST1")
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/embedding_ops.py", line 394, in embedding_lookup_v2
    return embedding_lookup(params, ids, "div", name, max_norm=max_norm)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/embedding_ops.py", line 328, in embedding_lookup
    transform_fn=None)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/embedding_ops.py", line 246, in _embedding_lookup_and_transform
    ret.set_shape(ids.get_shape().concatenate(element_shape_s))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py", line 1206, in set_shape
    if not self.shape.is_compatible_with(shape):
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py", line 1167, in shape
    self._tensor_shape = tensor_shape.TensorShape(self._shape_tuple())
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Ran out of memory in memory space vmem. It should not be possible to run out of vmem - please file a bug against XLA.

Largest program allocations in vmem:

  XLA label: register allocator spill slots
  Allocation type: scoped

  XLA label: %concatenate.724 = f32[3840,2]{0,1:T(2,128)} concatenate(f32[1,2]{0,1:T(2,128)}, f32[3,2]{0,1:T(2,128)}, f32[5,2]{0,1:T(2,128)}, f32[1,2]{0,1:T(2,128)}, ...(+2400)), dimensions={0}
  Allocation type: scoped

  XLA label: %concatenate.724 = f32[3840,2]{0,1:T(2,128)} concatenate(f32[1,2]{0,1:T(2,128)}, f32[3,2]{0,1:T(2,128)}, f32[5,2]{0,1:T(2,128)}, f32[1,2]{0,1:T(2,128)}, ...(+2400)), dimensions={0}
  Allocation type: scoped

  XLA label: %concatenate.724 = f32[3840,2]{0,1:T(2,128)} concatenate(f32[1,2]{0,1:T(2,128)}, f32[3,2]{0,1:T(2,128)}, f32[5,2]{0,1:T(2,128)}, f32[1,2]{0,1:T(2,128)}, ...(+2400)), dimensions={0}
  Allocation type: scoped

  XLA label: %concatenate.724 = f32[3840,2]{0,1:T(2,128)} concatenate(f32[1,2]{0,1:T(2,128)}, f32[3,2]{0,1:T(2,128)}, f32[5,2]{0,1:T(2,128)}, f32[1,2]{0,1:T(2,128)}, ...(+2400)), dimensions={0}
  Allocation type: scoped

2020-10-05 08:23:59.826142: W tensorflow/core/distributed_runtime/eager/remote_tensor_handle_data.cc:76] Unable to destroy remote tensor handles. If you are running a tf.function, it usually indicates some op in the graph gets an error: Ran out of memory in memory space vmem. It should not be possible to run out of vmem - please file a bug against XLA.

Largest program allocations in vmem:

  XLA label: register allocator spill slots
  Allocation type: scoped

  XLA label: %concatenate.724 = f32[3840,2]{0,1:T(2,128)} concatenate(f32[1,2]{0,1:T(2,128)}, f32[3,2]{0,1:T(2,128)}, f32[5,2]{0,1:T(2,128)}, f32[1,2]{0,1:T(2,128)}, ...(+2400)), dimensions={0}
  Allocation type: scoped

  XLA label: %concatenate.724 = f32[3840,2]{0,1:T(2,128)} concatenate(f32[1,2]{0,1:T(2,128)}, f32[3,2]{0,1:T(2,128)}, f32[5,2]{0,1:T(2,128)}, f32[1,2]{0,1:T(2,128)}, ...(+2400)), dimensions={0}
  Allocation type: scoped

  XLA label: %concatenate.724 = f32[3840,2]{0,1:T(2,128)} concatenate(f32[1,2]{0,1:T(2,128)}, f32[3,2]{0,1:T(2,128)}, f32[5,2]{0,1:T(2,128)}, f32[1,2]{0,1:T(2,128)}, ...(+2400)), dimensions={0}
  Allocation type: scoped

  XLA label: %concatenate.724 = f32[3840,2]{0,1:T(2,128)} concatenate(f32[1,2]{0,1:T(2,128)}, f32[3,2]{0,1:T(2,128)}, f32[5,2]{0,1:T(2,128)}, f32[1,2]{0,1:T(2,128)}, ...(+2400)), dimensions={0}
  Allocation type: scoped
```

**System information**
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
os: Linux
os kernel version: #1 SMP Debian 4.19.146-1 (2020-09-17)
os release version: 4.19.0-11-cloud-amd64
os platform: Linux-4.19.0-11-cloud-amd64-x86_64-with-debian-10.6
linux distribution: ('debian', '10.6', '')
linux os distribution: ('debian', '10.6', '')
mac version: ('', ('', '', ''), '')
uname: uname_result(system='Linux', node='shan-tf1', release='4.19.0-11-cloud-amd64', version='#1 SMP Debian 4.19.146-1 (2020-09-17)', machine='x86_64', processor='')
architecture: ('64bit', 'ELF')
machine: x86_64

- TensorFlow installed from (source or binary):
- TensorFlow version (use command below):

tf.version.VERSION = 2.3.0-dev20200620
tf.version.GIT_VERSION = v1.12.1-34769-gfd2d4cdb70
tf.version.COMPILER_VERSION = 7.3.1 20180303

- Python version:
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version:
- GPU model and memory:

      github.com/google/jax

      np.take and np.einsum aren't fused properly 3

        opened

        May 26, 2020

          AranKomat

        P2 (eventual)
        performance
        xla_issue

    I'm trying to translate [Product Key Memory ](https://arxiv.org/abs/1907.05242) …in PyTorch into JAX, and this requires the translation of [nn.EmbeddingBag](https://pytorch.org/docs/master/generated/torch.nn.EmbeddingBag.html) with per_sample_weights, as I haven't found any counterpart in JAX (but if you know, please let me know). For this, I wrote scatter_weighted_sum, the weighted sum version of scatter_add, in hope that it'll be efficient with fusing. However, jit didn't fuse np.take, reshape and np.einsum properly, which resulted in a huge intermediate object. Since #1979 concluded this sort of ops will be fused on GPU, I was wondering what is causing this problem. If by any chance this isn't supported on GPU, should this work on TPU? I'm using JAX ver. 0.1.67 with various Colab GPUs.

````python
hidden_dim = 512
n_keys = 512
batch = 2 ** 15
knn = 32
heads = 4
key = random.PRNGKey(0)
values = random.normal(key, (n_keys ** 2, hidden_dim))
indices = random.randint(key, (batch*heads, knn), 0, n_keys ** 2)
weights = random.normal(key, (batch*heads, knn))
@jit
def scatter_weighted_sum(inputs, indices, weights):
    num_bags = weights.shape[-1]
    dim = inputs.shape[-1]
    indices = indices.reshape(-1)
    tmp = inputs.take(indices, axis=0).reshape(-1, num_bags, dim)
    return np.einsum('ind, in -> id', tmp, weights)

````

Thanks,
Stefano

Context:
@kristen Is the MLIR team registered to this Dscourse instance or are they only in the LLVM MLIR discourse instance 3?
Cause generally we don’t have TF specific threads in the LLVM MLIR instance.


SIG Build January Meeting: January 11 @ 2pm

Question: SIG Build’s next meeting will be tomorrow, Tuesday, January 11, at 2pm Pacific time. Find the meeting details at bit.ly/tf-sig-build-notes 4, and feel free to suggest new agenda items.

Context:
Here’s a summary of some of the major points from the meeting.

Please fill out this form: SIG Build Monthly Meeting Time 2022. We’re considering a new meeting time every other month in for the sake of worldwide members for whom 2pm PST is not ideal.
I’ve merged configuration files that build TF’s Nightly test suite in Docker, and am working on changing our internal CI to use these containers.
Gentoo now packages ROCm!
Many of TF DevInfra team’s projects are delayed – manylinux2014 and Python 3.10 support are both stuck on build/test failures.

Our next meeting is February 1st at 2pm PST. See you then!


How to use TFJS for processing pen & paper filled assessments?

Question:
Hello, I had been use TFJS for a while for pose recognition (with PoseNet) and it is magnific! Now I wonder if I can use TF for a different use case: I want to scan printed assessments with multiple questions, where the options are circles that should be filled with a black pen by the user.
I imagine I need a way to recognize the frame containing all the answers, similar to the way QR-Codes corners are used… I appreciate any suggestion about what would be a good approach to this problem.
thanks!

Context:
Welcome to the forum and thanks for being part of the TensorFlow.js community!
A few things.

Glad you were enjoying PoseNet however please consider upgrading to MoveNet which is almost 100x faster and much more accurate. PoseNet was good when it came out, but MoveNet is now the standard   Learn more here:

      blog.tensorflow.org

Next-Generation Pose Detection with MoveNet and TensorFlow.js

  MoveNet is a human pose detection architecture developed at Google that is ultra fast and accurate. It was designed to detect difficult poses

So for your new problem I would first ask if you need machine learning for this task? Regular computer vision may be adequate depending on your data eg if its well scanned and just black and white it may be fairly trivial to find all black areas that are squares of a certain size and which of those contain more filled pixels than others for example.

That being said if you do want to use machine learning you will probably want to retrain some sort of object detection model to find objects of interest and their positions eg a filled box vs an unfilled box etc.
For that I highly recommend following this great tutorial by Hugo:

      blog.tensorflow.org

Custom object detection in the browser using TensorFlow.js

  Train a custom MobileNetV2 using the TensorFlow 2 Object Detection API and Google Colab for object detection, convert the model to TensorFlow.js

You can then run that resulting trained model in the browser to detect custom objects like you need and find their locations in the given image.


Adopting Open-Source Dockerfiles for Official tf-nightly CI

Question:
The TensorFlow OSS DevInfra Team and TF SIG Build are developing new Dockerfiles in the SIG Build GitHub repo 8 that we want to be used for all of TensorFlow’s official build and test environments. They are published to SIG Build on DockerHub 5.  Our first milestone is to use the Dockerfiles to build the TF Nightly packages with the following goals:

Container-built packages are functionally identical to the current package
Developers (you!) can build the same packages that we do with minimal effort

That milestone is ready for verification. I’ve set up internal CI jobs that use the containers to build tf-nightly packages that are very similar to the current ones, and I’d like your help to evaluate them for functional differences. Starting on Monday the 30th, we’ve been using the containers to build our official tf-nightly packages.
Here is a set of packages we built at the same commits for advance comparison. There are minor cosmetic differences but we’d like your help to find out if there are any functional differences between packages on the same row of the table below.

Short Git Hash
Old Non-Docker Builds
New Docker Builds

5af3afc559
GPU Python 3.9
GPU Python 3.9 3

5af3afc559
GPU Python 3.8
GPU Python 3.8

5af3afc559
GPU Python 3.7
GPU Python 3.7

1d51452b18
CPU Python 3.9
CPU Python 3.9 1

1d51452b18
CPU Python 3.8
CPU Python 3.8 1

1d51452b18
CPU Python 3.7
CPU Python 3.7

Here’s how you can help us make the containers useful for you:

Install and compare the sample packages above. If you compare the two wheels for any of the rows, do they have any differences that would affect your workflow?
Check out the containers on DockerHub 5 and the tf-nightly build instructions at the SIG Build repository 8. Are you able to build TensorFlow with them? If you use the same git hashes as above, how is your package different?
With the new packages that came out starting on Nov. 30, is anything different about them in a way that affects your workflow?

Please give all feedback in this thread. Thank you for your help!

Context:
If you have Docker (and nvidia-docker 1 if you want to run GPU TensorFlow) set up already, here’s how to test out one of the packages linked in the OP from inside the containers:
CPU:
docker pull tensorflow/build:latest-python3.9
docker run -it --rm tensorflow/build:latest-python3.9 bash
wget https://storage.googleapis.com/tensorflow-nightly/prod/tensorflow/nightly_release/ubuntu_tfdkr/cpu_py39/6/20211117-000455/pkg/tf_nightly_cpu-2.8.0.dev20211117-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
pip install ./tf_nightly*
python
import tensorflow as tf

GPU with nvidia-docker:
docker pull tensorflow/build:latest-python3.9
docker run --gpus=all -it --rm tensorflow/build:latest-python3.9 bash
wget https://storage.googleapis.com/tensorflow-nightly/prod/tensorflow/nightly_release/ubuntu_tfdkr/gpu_py39/6/20211117-000458/pkg/tf_nightly-2.8.0.dev20211117-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
pip install ./tf_nightly*
python
import tensorflow as tf
tf.config.list_physical_devices('GPU')


How to add multiple pre-processing steps and a post-processing step for text-classifiction model to serve via tensorflow-serving?

Question:
What I currently have and trying to do:
When I receive a request from a client to the model in tensorflow-serving,  I first need to process them using 13 regexes, then pass that text through tf.keras.preprocessing.text.Tokenizer 1 to convert them to numbers(or token), and then pass them to tf.keras.preprocessing.sequence.pad_sequences to add 0s (for the sentences whose lenght doesn’t match the input that the model expects) at the end of each array(for a batch of inputs), then this(a single sentence or a batch of sentences as tokens) will be fed to a tf.keras model to get some probabilities as outputs. And I then need to map these probabilites(different thresholds for different units) to texts and return it to the client.

What problems am I currently facing trying to accomplish above:
While trying to put together all that to be able to serve the model using tensorflow-serving, I learned that some parts can be converted to tensorflow functions, but not all of it.

regexes: I still couldn’t figure out where and how to put my regexes to be able to manipulate the text.
tokenizer: I learned from some blogs and SO questions, that tf.lookup.StaticHashTable can be used for this purpose.
pad_sequences: no help with this too.
post-processing: I could find very little information to do this.

I read the beginner and advanced blogs on tensorflow-transform tutorials page, but either of them mentioned how to link those tft functions to the tf-keras model, while saving it. And I could also find some information about adding pre-processing for serving, but all of them involved tensorflow code and some workarounds, but they didn’t cover what I am trying to achieve even indirectly.
I can provide more information as required.
How do I add these steps to the graph, while saving the model?

Context:
1 & 3 & 4. After training the model, you can save graph with pre-processing and post-processing steps like below
...
...

# some training steps
model = ...
model.compile(...)
model.fit(...)

@tf.function
def inference_function(text):
  # do some preprocessing
  text = tf.strings.regex_replace(text, # ... some regex patterns...)
  token_ids, starts, ends = tokenizer.tokenize_with_offsets(text)

  model_inputs = # prepare model inputs using token_ids

  # inference model
  model_outputs = model(model_inputs)

  outputs = # do some post-processing with starts, ends, and model_outputs
  return outputs

# https://www.tensorflow.org/api_docs/python/tf/keras/Model#save
model.save(
  "some path to save the model",
  signatures={
    "inference_fn": inference_function.get_concrete_function(tf.TensorSpec([None], dtype=tf.string)),
  }
)

Yes! After training the sentencepiece model, you can load and use it with text.SentencepieceTokenizer 1 in TF graph.


Converting PyTorch to Keras: Internal Blocks Not Showing Up

Question:
Hi everyone,
For a personal project I’m trying to recreate the NBeats architecture into Keras, and I don’t think I’m doing it correctly but am not sure why.
The page I’m working off of as a ground truth can be found here:  https://github.com/ElementAI/N-BEATS/blob/master/models/nbeats.py
Here’s the starter PyTorch code that I’m trying to convert:
class NBeatsBlock(t.nn.Module):
    def __init__(self,
                 input_size,
                 theta_size: int,
                 basis_function: t.nn.Module,
                 layers: int,
                 layer_size: int):
        super().__init__()
        self.layers = t.nn.ModuleList([t.nn.Linear(in_features=input_size, out_features=layer_size)] +
                                      [t.nn.Linear(in_features=layer_size, out_features=layer_size)
                                       for _ in range(layers - 1)])
        self.basis_parameters = t.nn.Linear(in_features=layer_size, out_features=theta_size)
        self.basis_function = basis_function

    def forward(self, x: t.Tensor) -> Tuple[t.Tensor, t.Tensor]:
        block_input = x
        for layer in self.layers:
            block_input = t.relu(layer(block_input))
        basis_parameters = self.basis_parameters(block_input)
        return self.basis_function(basis_parameters)

class NBeats(t.nn.Module):
    def __init__(self, blocks: t.nn.ModuleList):
        super().__init__()
        self.blocks = blocks

    def forward(self, x: t.Tensor, input_mask: t.Tensor) -> t.Tensor:
        residuals = x.flip(dims=(1,))
        input_mask = input_mask.flip(dims=(1,))
        forecast = x[:, -1:]
        for i, block in enumerate(self.blocks):
            backcast, block_forecast = block(residuals)
            residuals = (residuals - backcast) * input_mask
            forecast = forecast + block_forecast
        return forecast

class GenericBasis(t.nn.Module):
    def __init__(self, backcast_size: int, forecast_size: int):
        super().__init__()
        self.backcast_size = backcast_size
        self.forecast_size = forecast_size

    def forward(self, theta: t.Tensor):
        return theta[:, :self.backcast_size], theta[:, -self.forecast_size:]

Here’s  the Keras code I have to translate:
class NBeatsBlock(keras.layers.Layer):
    def __init__(self,
                 theta_size: int,
                 basis_function: keras.layers.Layer,
                 layer_size: int = 4):
        super(NBeatsBlock, self).__init__()
        self.layers_          = [keras.layers.Dense(layer_size, activation = 'relu')
                                    for i in range(layer_size)]
        self.basis_parameters = keras.layers.Dense(theta_size)
        self.basis_function   = basis_function

    def call(self, inputs):
        x = self.layers_[0](inputs)
        for layer in self.layers_[1:]:
            x = layer(x)
        x = self.basis_parameters(x)
        return self.basis_function(x)

class NBeats(keras.layers.Layer):
    def __init__(self,
                 blocksize: int,
                 theta_size: int,
                 basis_function: keras.layers.Layer):
        super(NBeats, self).__init__()
        self.blocks = [NBeatsBlock(theta_size =  theta_size, basis_function =  basis_function) for i in range(blocksize)]

    def call(self, inputs):
        residuals = K.reverse(inputs, axes = 0)
        forecast  = inputs[:, -1:]
        for block in self.blocks:
            backcast, block_forecast = block(residuals)
            residuals                = residuals - backcast
            forecast                 = forecast + block_forecast
        return forecast

class GenericBasis(keras.layers.Layer):
    def __init__(self, backcast_size: int, forecast_size: int):
        super().__init__()
        self.backcast_size = backcast_size
        self.forecast_size = forecast_size

    def call(self, inputs):
        return inputs[:, :self.backcast_size], inputs[:, -self.forecast_size:]

If I try and make a model from the Keras code it works, but I don’t think it’s constructed correctly.
Here’s a simple model:
inputs       = Input(shape = (1, ))
nbeats       = NBeats(blocksize = 4, theta_size = 7, basis_function = GenericBasis(7, 7))(inputs)
out          = keras.layers.Dense(7)(nbeats)
model        = Model(inputs, out)
My concern is that the internal NBeatsBlock layers are not actually being used in the model I just created.
My model summary reads like this:
,
And as you can see there’s nothing that indicates the internal Dense layers are there.
And if I plot the model I get the following diagram:

So I don’t think I’m doing things correctly but I’m also not sure where I’m going wrong with how I’m constructing it.  I’m guessing there are small differences in how PyTorch & Keras work that I’m not picking up on.

Context:
I’ve not personally verified the correct implementation  but there was a parallel Pytorch and Keras impl at:

      GitHub

GitHub - philipperemy/n-beats: Keras/Pytorch implementation of N-BEATS:... 1

  Keras/Pytorch implementation of N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. - GitHub - philipperemy/n-beats: Keras/Pytorch implementation of N-BEATS: Neural ...


Help training MobileNetV3Small model on custom image classification

Question:
Hi everyone, I’m new to the Tensorflow Keras API, and thought I would use it with the tensorflow-metal plugin from Apple to train a custom MobileNetV3Small model on my M1 Pro MacBook for the task of image classification. This is for my app DeTeXt, that classifies drawings into LaTeX symbols. Currently I’m using a MobileNetV2 model that I had trained on a GPU cluster using the PyTorch API (code here 1).
Here is the code I use to train my custom network from scratch on the images I have:
import tensorflow as tf
import pdb

EPOCHS = 5
BATCH_SIZE = 128
LEARNING_RATE = 0.003
SEED=1220

if __name__ == '__main__':
	# Load train and validation data
	train_ds = tf.keras.preprocessing.image_dataset_from_directory(
								'/Volumes/detext/drawings/',
							    color_mode="grayscale",
  								seed=SEED,
  								batch_size=BATCH_SIZE,
								labels='inferred',
								label_mode='int',
								image_size=(200,300),
								validation_split=0.1,
								subset='training')

	val_ds = tf.keras.preprocessing.image_dataset_from_directory(
								'/Volumes/detext/drawings/',
							    color_mode="grayscale",
  								seed=SEED,
  								batch_size=BATCH_SIZE,
								labels='inferred',
								label_mode='int',
								image_size=(200,300),
								validation_split=0.1,
								subset='validation')

	# Get the class names
	class_names = train_ds.class_names
	num_classes = len(class_names)

	# Create model
	model = tf.keras.applications.MobileNetV3Small(
    	input_shape=(200,300,1), alpha=1.0, minimalistic=False,
    	include_top=True, weights=None, input_tensor=None, classes=num_classes,
    	pooling=None, dropout_rate=0.2, classifier_activation="softmax",
    	include_preprocessing=True)

	# Compile model
	model.compile(
		   optimizer=tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE),
           loss=tf.keras.losses.SparseCategoricalCrossentropy(),
           metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])

    # Training
	model.fit(train_ds, epochs=EPOCHS, validation_data=val_ds)

	model.save('./saved_model3/')

While the training runs smooth and fast with the metal plugin, the validation accuracy is very low after 5 epochs, and I suspect it is either predicting the same class every time, or there is an error somewhere in my setup above. I have tried rescaling the inputs myself (and removing rescaling layer from model), but no matter what I try, the validation accuracy it outputs is really low. Here is the output (warnings and all) after 2 epochs:
Found 210454 files belonging to 1098 classes.
Using 189409 files for training.
Metal device set to: Apple M1 Pro

systemMemory: 32.00 GB
maxCacheSize: 10.67 GB

2021-12-16 10:02:46.369476: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2021-12-16 10:02:46.369603: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
Found 210454 files belonging to 1098 classes.
Using 21045 files for validation.
Epoch 1/2
2021-12-16 10:02:50.610564: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2021-12-16 10:02:50.619328: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2021-12-16 10:02:50.619628: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
1480/1480 [==============================] - ETA: 0s - loss: 1.7621 - sparse_categorical_accuracy: 0.57022021-12-16 10:12:58.720162: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
1480/1480 [==============================] - 626s 422ms/step - loss: 1.7621 - sparse_categorical_accuracy: 0.5702 - val_loss: 9.5837 - val_sparse_categorical_accuracy: 0.0052
Epoch 2/2
1480/1480 [==============================] - 622s 420ms/step - loss: 1.0791 - sparse_categorical_accuracy: 0.6758 - val_loss: 7.3651 - val_sparse_categorical_accuracy: 0.0423
2021-12-16 10:23:40.260143: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
/Users/venkat/miniforge3/envs/tf-metal/lib/python3.9/site-packages/keras/utils/generic_utils.py:494: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  warnings.warn('Custom mask layers require a config and must override '

For reference, I was getting validation micro-F1 (which is same as accuracy) of over 60% with MobilenetV2 in PyTorch. Anyone have any idea what I’m doing wrong here?

Context:
The issue seems to be specific to certain types of operations/layers in Tensorflow, and specifically with respect to the validation accuracy (similar to this issue 2). When I build my own custom model with convolutions like so:
    model = Sequential([
  layers.Rescaling(1./255, input_shape=(IMG_HEIGHT, IMG_WIDTH, 1)),
  layers.Conv2D(16, 1, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 1, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(64, 1, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Dropout(0.2),
  layers.Flatten(),
  layers.Dense(128, activation='relu'),
  layers.Dense(num_classes),
  layers.Softmax()
])

training proceeds as expected, with a high validation accuracy as well. Below is the output for the above model:
Found 210454 files belonging to 1098 classes.
Metal device set to: Apple M1 Pro

systemMemory: 32.00 GB
maxCacheSize: 10.67 GB

2021-12-21 12:27:24.005759: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2021-12-21 12:27:24.006206: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
Found 210454 files belonging to 1098 classes.
Using 31568 files for validation.
2021-12-21 12:27:26.965648: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2021-12-21 12:27:26.968717: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2021-12-21 12:27:26.969214: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
1645/1645 [==============================] - ETA: 0s - loss: 2.1246 - sparse_categorical_accuracy: 0.52732021-12-21 12:32:57.475358: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
1645/1645 [==============================] - 353s 214ms/step - loss: 2.1246 - sparse_categorical_accuracy: 0.5273 - val_loss: 1.3041 - val_sparse_categorical_accuracy: 0.6558
2021-12-21 12:33:19.600146: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.

However, the very same code with the MobileNetV3Small model (instead of my custom model) produces the following output:
Found 210454 files belonging to 1098 classes.
Metal device set to: Apple M1 Pro

systemMemory: 32.00 GB
maxCacheSize: 10.67 GB

2021-12-21 12:34:46.754598: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2021-12-21 12:34:46.754793: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
Found 210454 files belonging to 1098 classes.
Using 31568 files for validation.
2021-12-21 12:34:49.742015: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2021-12-21 12:34:49.747397: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2021-12-21 12:34:49.747606: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
1645/1645 [==============================] - ETA: 0s - loss: 2.4072 - sparse_categorical_accuracy: 0.46722021-12-21 12:41:28.137948: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
1645/1645 [==============================] - 415s 252ms/step - loss: 2.4072 - sparse_categorical_accuracy: 0.4672 - val_loss: 21.6091 - val_sparse_categorical_accuracy: 0.0131
2021-12-21 12:41:46.017580: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
/Users/venkat/miniforge3/envs/tf-metal/lib/python3.9/site-packages/keras/utils/generic_utils.py:494: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  warnings.warn('Custom mask layers require a config and must override '

The validation loss/accuracy is hilariously bad, and I find that the model constantly predicts the same class. My guess is that MobileNetV3Small seems to contain some operations/layers that don’t work well with tensorflow-metal for whatever reason, and only Apple Engineers can fix this problem at a low level.


SIG Addons December Meeting Moved to 12/16

Question:
Hi All,
Just wanted to call attention to our meeting being moved to 12/9 this month. This will be our first back log grooming meeting so looking forward to your attendance!

Context: Unfortunately I have a late conflict and calendar should be moved to 12/16. Sincerely apologize.


SIG Build December Meeting: December 7 @ 2pm

Question:
SIG Build’s next meeting will be tomorrow, Tuesday, December 7, at 2pm Pacific time. Find the meeting details at bit.ly/tf-sig-build-notes 4, and feel free to suggest new agenda items.
One of the big discussion topics will be the Docker containers, which are discussed here: Adopting Open-Source Dockerfiles for Official tf-nightly CI 1

Context:
Thanks to everyone who attended yesterday. Here’s a summary of the biggest topics:

Next month’s meeting will be January 11th. I have not moved the calendar slot yet.
We discussed the Docker containers. Notable points: @angerson will work on a public roadmap; there are some permissions-related tests that should not exist; GPU passthrough isn’t working right now; the resulting wheels do seem to be the same; there are many issues with cache misses that we know about but are low priority.
DevInfra is aware of Numpy’s Python 3.7 deprecation plans and need to discuss with the TF product team about what we’re going to do.


Question:
Hi there,
I am curious about TFG’s designed representation of tensors. From the code, I found that tensor is represented as an attribute of the tfg.op.  The code is like below:
// tensorflow/core/ir/importexport/convert_tensor.h
// Converts an TensorFlow tensor proto into an MLIR elements attribute.
tensorflow::StatusOr<ElementsAttr> ConvertTensorProto(
    const tensorflow::TensorProto& input_tensor, Builder builder,
    TFGraphDialect* tfgDialect);

From the above code, a tensorflow tensor is converted to mlir::ElementsAttr for a specific op.  This would be an attribute for the op node.
However, MLIR attributes 1 read following:

attributes are compile-time known constant values

So, from my understanding,  tfg does not allow modifying the tensor’s data, because the tensor attribute should be constant value.
In that case, Are the following understanding correct?

tfg by design does not allow optimizations like constant_folding, which  would modify tensor’s data. (Or have to recreate a new node and replace the old one).
Why not create a  type instead of attribute in TFG MLIR? As type would allow mutable part 1 listed in the link.
Similar design appreas on ShapeAttr , it also does not have the setShape method, meaning this by design could not be modified.

As tfg is designed to replace grappler, in grappler, it could easily change those things in NodeDef, how could tfg do these things? Should tfg always create a new node and replace the old one?

Context:
Stonepia:

So I have to attach a pointer-like attribute, and mutate that underlying information, is this correct?

You really can’t mutate Type and Attribute safely: they are stored in a map and hashed. Every operation (node…) using a given type will reuse the same instance. When you “recreate” a Type what happens is actually hashing and lookup to see if it already exists, in which case it gets returned.
In general, if you need to just compute transient information in a transformation, you may not store them directly by mutating the IR, you may keep a map on the side from operations to “split_strategy” and use this as your temporary storage. It does not prevent you from materializing the “split_strategy” as type/attribute annotation when you’re done, but that shouldn’t be too heavy any more at this point.


[AutoKeras] Bug in using tf.Dataset for training

Question:
Hi everyone!
I was exporting a custom data generator to a tf.Dataset to use my dataset in a memory-efficient way. However, I am encountering this error:-
ValueError: Expect x to be a non-empty array or dataset.

To keep things clean here, I have put all the heavy information on a GitHub Thread 3.
If anyone requires additional information or help for reproduction, please do not hesitate to ping me! As I have mentioned in my issue, using torch.random.radn((...)) for a dummy dataset seems the fastest way
Dissecting the error message, it seems that in train.py (keras/engine) the Dataset it gets is empty (even though its not) which yields no updates to the model parameters. Since no updates are made, no logs are created. logs is unchanged to its OG value None and it hits the raise statement.
If anyone has any idea, please do help me out!

Context:
I made a little progress; but help is really appreciated as Im kinda confused
The issue seems to be directly at autokeras/auto_model.py at 78223dfc63056414e32e202b158dbc981be48dc9 · keras-team/autokeras · GitHub 3 just when I pass the tf.Dataset - its simply empty. I have checked numerous times but can confirm I am not passing an empty Dataset. strange. but I will surely double check.
Again, if anyone can expedite this process for me - that would be really appreciated!


Standardize Github community health files in the Ecosystem

Question:
Hi,
I would like to collect a little some feedback about the idea of standardizing a little bit a new candidate contributor experience across SIGs as we are moving forward a multi repositories/SIGs oriented ecosystem (e.g. see the new TF-micro and Keras standalone repos or TF core as product RFC 7).
With this I hope that we could discuss a minimal set of common contents for the two main Github community health files 4.
I have currently submitted two DRAFT PR to collect feedback and comments:

README.md 2

CONTRIBUTING.md 4

At the end of the process, if we could find a consensus on the minimal set of info to maintain, then every SIG could also add extra sections in the footers or links to other Markdown files available in its own repository.
I hope that we could lower the cognitive overhead of a candidate contributor navigating over the ecosystem.
For general comments we can use this thread to discuss the topic.
Thanks
/cc @thea @Joana @ewilderj @yarri-oss

Context: Thanks for starting this thread. I’d like to make sure we give special attention to the community Maintainers. We have a mix of very active vs. more passive maintainers across the SIGs – both types are important and to be encouraged! I would like to propose adding a CALL_FOR_MAINTAINERS.md file to your list above.