Not able to lower tf.sets.intersection to HLO

Question:
My python code snippet is like:
import os
import numpy as np
import tensorflow as tf
from tensorflow.python.eager import context
from tensorflow.python.framework import dtypes
from tensorflow.python.platform import test
from tensorflow.python.framework import config
import tensorflow.compat.v1 as tf
config.enable_mlir_bridge()
tf.config.experimental.enable_mlir_bridge()
class CustomModule(tf.Module):
def init(self):
super(CustomModule, self).init()
self.condition = tf.Variable(np.array([[True, False, False],[False, True, False],[True, True, True]]), dtype = tf.bool)
self.x = tf.Variable(np.array([[1, 2, 3],[4, 5, 6],[7, 8, 9]]), dtype = tf.int32)
self.y =tf.Variable(np.array([[11, 12, 13],[14, 15, 16],[17, 18, 19]]), dtype = tf.int32)
@tf.function
def call(self, x):
r = tf.where(self.condition, self.x, self.y)
m= tf.where(self.condition, self.x, self.y)
c=tf.sets.intersection(tf.expand_dims(r, 0),tf.expand_dims(m, 0))

return c

module = CustomModule()
module_with_signature_path = os.path.join("/data/aruna/tf_ops", ‘sets_intersection’)
call = module.call.get_concrete_function(tf.TensorSpec(shape=(), dtype=tf.int32))
signatures = {‘predict’: call}
tf.saved_model.save(module, module_with_signature_path, signatures=call)
print(‘Saving model…’)
if name == ‘main’:
test.main()

I ran this python code and got saved_model.pb.
Then I used following commands:
tensorflow/compiler/mlir/tf-mlir-translate --savedmodel-objectgraph-to-mlir --tf-savedmodel-exported-names=predict  -tf-enable-shape-inference-on-import=true $PWD -o sample.mlir
tensorflow/compiler/mlir/tf-opt --tf-executor-to-functional-conversion --tf-shape-inference -xla-legalize-tf  --print-ir-before-all  sample.mlir
TF dialect looks like:
// -----// IR Dump Before LegalizeTF //----- //
builtin.func private @__inference___call___750(%arg0: tensor {tf._user_specified_name = “x”}, %arg1: tensor<!tf_type.resource>, %arg2: tensor<!tf_type.resource>, %arg3: tensor<!tf_type.resource>) → (tensor<?x3xi64>, tensor<?xi32>, tensor<3xi64>) attributes {tf._construction_context = “kEagerRuntime”, tf._input_shapes = [#tf_type.shape<>, #tf_type.shape<>, #tf_type.shape<>, #tf_type.shape<>], tf.signature.is_stateful} {
%cst = “tf.Const”() {device = “”, value = dense<0> : tensor} : () → tensor
%cst_0 = “tf.Const”() {device = “”, value = dense<0> : tensor}2021-09-08 09:56:50.579733: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
: () → tensor
%0 = “tf.ReadVariableOp”(%arg2) {device = “”} : (tensor<!tf_type.resource>) → tensor<3x3xi32>
%1 = “tf.ReadVariableOp”(%arg2) {device = “”} : (tensor<!tf_type.resource>) → tensor<3x3xi32>
%2 = “tf.ReadVariableOp”(%arg3) {device = “”} : (tensor<!tf_type.resource>) → tensor<3x3xi32>
%3 = “tf.ReadVariableOp”(%arg3) {device = “”} : (tensor<!tf_type.resource>) → tensor<3x3xi32>
%4 = “tf.ReadVariableOp”(%arg1) {device = “”} : (tensor<!tf_type.resource>) → tensor<3x3xi1>
%5 = “tf.Select”(%4, %0, %2) {device = “”} : (tensor<3x3xi1>, tensor<3x3xi32>, tensor<3x3xi32>) → tensor<3x3xi32>
%6 = “tf.ExpandDims”(%5, %cst) {device = “”} : (tensor<3x3xi32>, tensor) → tensor<1x3x3xi32>
%7 = “tf.ReadVariableOp”(%arg1) {device = “”} : (tensor<!tf_type.resource>) → tensor<3x3xi1>
“tf.NoOp”() {_acd_function_control_output = true, device = “”} : () → ()
%8 = “tf.Select”(%7, %1, %3) {device = “”} : (tensor<3x3xi1>, tensor<3x3xi32>, tensor<3x3xi32>) → tensor<3x3xi32>
%9 = “tf.ExpandDims”(%8, %cst_0) {device = “”} : (tensor<3x3xi32>, tensor) → tensor<1x3x3xi32>
%10:3 = “tf.DenseToDenseSetOperation”(%6, %9) {T = i32, device = “”, set_operation = “intersection”, validate_indices = true} : (tensor<1x3x3xi32>, tensor<1x3x3xi32>) → (tensor<?x3xi64>, tensor<?xi32>, tensor<3xi64>)
%11 = “tf.Identity”(%10#0) {device = “”} : (tensor<?x3xi64>) → tensor<?x3xi64>
%12 = “tf.Identity”(%10#1) {device = “”} : (tensor<?xi32>) → tensor<?xi32>
%13 = “tf.Identity”(%10#2) {device = “”} : (tensor<3xi64>) → tensor<3xi64>
return %11, %12, %13 : tensor<?x3xi64>, tensor<?xi32>, tensor<3xi64>
}
Error is:
sample.mlir:5:3: error: The following operations cannot be legalized: tf.DenseToDenseSetOperation (count: 1); tf.NoOp (count: 1); tf.ReadVariableOp (count: 6). These legalization failure(s) may be due to missing TF to HLO lowerings and/or unsupported attributes, etc.
builtin.func private @__inference___call___340(%arg0: tensor {tf._user_specified_name = “x”}, %arg1: tensor<!tf_type.resource>, %arg2: tensor<!tf_type.resource>, %arg3: tensor<!tf_type.resource>) → (tensor<?x3xi64>, tensor<?xi32>, tensor<3xi64>) attributes {tf._construction_context = “kEagerRuntime”, tf._input_shapes = [#tf_type.shape<>, #tf_type.shape<>, #tf_type.shape<>, #tf_type.shape<>], tf.signature.is_stateful} {
^
sample.mlir:5:3: error: Emitting more detail about one op that failed to legalize…
builtin.func private @__inference___call___340(%arg0: tensor {tf._user_specified_name = “x”}, %arg1: tensor<!tf_type.resource>, %arg2: tensor<!tf_type.resource>, %arg3: tensor<!tf_type.resource>) → (tensor<?x3xi64>, tensor<?xi32>, tensor<3xi64>) attributes {tf._construction_context = “kEagerRuntime”, tf._input_shapes = [#tf_type.shape<>, #tf_type.shape<>, #tf_type.shape<>, #tf_type.shape<>], tf.signature.is_stateful} {
^
sample.mlir:20:61: error: ‘tf.DenseToDenseSetOperation’ op is not legalizable
%outputs_23:3, %control_24 = tf_executor.island wraps “tf.DenseToDenseSetOperation”(%outputs_14, %outputs_21) {T = i32, device = “”, set_operation = “intersection”, validate_indices = true} : (tensor<1x3x3xi32>, tensor<1x3x3xi32>) → (tensor<?x3xi64>, tensor<?xi32>, tensor<3xi64>)

Context:
tf.CropAndResize
tf.StridedSlice
tf.Unique
tf.Where
tf.SparseToDense
tf.NonMaxSuppressionV4
tf.TensorListFromTensor
tf.TensorListGetItem
tf.DenseToDenseSetOperation
tf.TensorListReserve
tf.TensorListSetItem
tf.TensorListStack
tf.TopKV2
For the above ops also I am getting same error while lowering these ops to HLO


Question:
Hello
for keras platform in model.compile only we use single learning rate, but I need multi learning rate for my model. for example my model include, Backbone with learning rate of 10^-4 and one transformer with learning rate of 10^-3. How could I set this two learning rate inside the model.compile with Adam optimizer or any optimizer?

Context: It seams that TensorFlow Addons has an optimizer with this capability: tfa.optimizers.MultiOptimizer  |  TensorFlow Addons 10


Question:
As we are updating LLVM two times at day 5 I’ve tried to query bazel with:
bazel aquery  "rdeps(//tensorflow:*,//third_party/llvm:*)" --include_aspects 2>/dev/null | grep Compiling | wc -l
I am not a bazel ninja so probably the query could be wrong or improved but I currently see 9938 files on master (CPU only).
What is the effect of this bi-daily rolling update on the average community contributor compiling workflow/environment and his bazel cache?

Context:
I’d assume that Bazel is smart enough to only recompile the pieces that actually changed in there, so the impact will vary depending on the actual updates we’re picking up.
As a workflow, when I use to develop in LLVM on a laptop, I would have a cron script that would run a git pull and build at 7am before I show up in the office so that when I arrive I have the most recent copy of the code with the build cache up-to-date


Question:
Hi,
We’re calling SavedModelBundle.exporter("/data/").withSession(sess).withSignature(mySig).export to save a model. It seems to save out fine (according to the cli tool). But when we’re loading it from the Python side using tf.saved_model.load("/data/"), we’re getting the following error in ValueError: Feeds must be tensors.
Any ideas ?
more details…
Signature (according to saved_model_cli)
  The given SavedModel SignatureDef contains the following input(s):
    inputs['_input_0'] tensor_info:
        dtype: DT_FLOAT
        shape: (10)
        name: Placeholder_1:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['_out_0'] tensor_info:
        dtype: DT_FLOAT
        shape: (10)
        name: Relu_1:0
  Method name is:

And full python stacktrace
wrap_function.py.prune(/apps/python3/lib/python3.7/site-packages/tensorflow/python/eager/wrap_function.py:262)
load_v1_in_v2.py._extract_saver_restore(/apps/python3/lib/python3.7/site-packages/tensorflow/python/saved_model/load_v1_in_v2.py:105)
load_v1_in_v2.py.load(/apps/python3/lib/python3.7/site-packages/tensorflow/python/saved_model/load_v1_in_v2.py:211)
load_v1_in_v2.py.load(/apps/python3/lib/python3.7/site-packages/tensorflow/python/saved_model/load_v1_in_v2.py:263)
load.py.load_internal(/apps/python3/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py:613)
load.py.load(/apps/python3/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py:578)

Context:
What version of TF-Java exported the model, and what version of TF Python is loading it in?
It’s a little weird it’s coming in as a TF v1 model, we might need to set a bit somewhere to have it load using the appropriate path.


Training CNN with Keras and 32FC1 matrix dataset

Question:
Good evening, my problem is that I want to train a Keras CNN that could tell me if in a image there is a sewer or not.
I have 2 datasets (one with positives images with sewer and another with no sewer) files with a 8000x60 matrix of decoded depth images, each image if 80x60 so each dataset has like  100 images.
My problem is that i dont know how to code that input to train the CNN. I have always worked with png datasets and now that type. If you have questions just ask.
Thanks in advance.

Context: If your images are already decoded into a matrix, you can try to use tf.data.Dataset.from_tensor_slices() method (tf.data.Dataset  |  TensorFlow Core v2.6.0 2) to create inputs for your model. You pass a tuple into this method: the first element is decoded image matrix (could be numpy array or other array-like types), the second element is at array of integer labels (0 and 1 in your case).


Keras: slow startup for large models due to metric-initialization

Question:
When creating large models (couple thousands nodes) in graph mode, initializing the metrics can take a very long time. The following toy example takes ~30 seconds on my machine (TF 2.6) to start training:
import tensorflow as tf
import numpy as np
from tensorflow.python.keras import backend as K

with K.get_session() as sess:
    print("DEF")
    model = tf.keras.Sequential(
        [tf.keras.layers.Dense(1) for _ in 500]
    )
    print("METRICS")
    metrics = [tf.keras.metrics.Accuracy(str(i)) for i in range(100)]

    print("COMPILE")
    model.compile(loss="mse", metrics=metrics, run_eagerly=False)
    x, y = np.zeros((2, 1000), dtype=np.float32)
    print("FIT")
    model.fit(x=x, y=y)

Most of the startup time is spend in this loop 6 initializing the metrics.
In the actual model I am currently investigating, startup takes ~20 minutes since it’s quite a large model with data loading included in the graph and ~400 metrics. The latter is due to having 4 per-class metrics for ~100 classes. This time quadruples when adding another GPU with MirroredStrategy. What could I do to improve startup time in this case? So far, I’ve tried:

running in eager mode, which works fine on a single GPU, but scaling out is going to be more challenging
Creating one metric-class for all classes so that I only need to register 4 metrics. But it doesn’t seem to be possible for metrics to return arrays.

Context: Turns out it’s only a problem with Tensorflow 1.x graph mode. Removing the line with K.get_session() as sess: fixes it.


Are there any types of Neural Network which is not possible built by Keras platform? and why?

Question:
By Considering these Chart of Complete Neural Networks Types:

      ImgBB

Network-Types 8

  Image Network-Types hosted in ImgBB

And also here is a Commonly used Neural Network types in Keras I found, which I take note also for its most Practice Cases (You can correct me if I am wrong):
1 DNN (Deep Neural Network) → Most Practice Cases: Time series + predicting plain response y variable (flexible on numeric integer categoric
2 CNN (Convolution Neural Network) → Most Practice Cases: Time Series + Image Classification/Computer Vision
3 LSTM (Long Short-Term Memory) → Most Practice Cases: Time Series + NLP, → Recommended use for a long set of training data (More complex structure than GRU → LSTM has three gates (namely input, output and forget gates)
4 RNN (Recurrent Neural Network) → Most Practice Cases: Time Series + NLP, RNN → Recommended use for Sequence Data (faster training, computationally less expensive
5 GRU (Gated Recurrent Unit → Most Practice Cases: Time Series + NLP GRU → Recommended use for a shorter set of training data (Less complex structure than LSTM GRU has two gates (reset and update gates)
6 Auto Encoders (AE) → Most Practice Cases: Image Classification/Computer Vision (Noising & Denoising), Auto Encoders basically are a set of CNN, Pooling2D, and CNN-Transpose
Finally my Question:

Are there any types of Neural Network in above chart which structure of Network are currently not possible to build by Keras component?

if there’s any Network which aren’t possible, could you point me what types, and why?

Are there any more Commonly used Neural Network aside from what I notes above? Appreciate it if theres any improvisation added to it

Appreciate any effort put into this question, Thank You!

Context:
Hi Jovan,
I’d associate  GRU, RNN and LSTM with sequences instead of time series. It’s broader and easier to remember. That’s why NLP is in the same group (sequence of characters/words)
As far as I know, Keras can build all the Neural Networks presented and if the layer you need is not available directly, you can customize or create your own with the same API.
The only detail on your image that I could spot is the Support Vector Machine (SVM). I don’t know if that can be considered a Neural Network and I don’t know if that can be built using Keras.


Question:
Hi folks:
I havn’t done any TFJS since version 2.3.0, looks like we are now at 3.8.0.
Just want to introduce myself.  My name is Jeremy Ellis, social media @rocksetta. I have spent almost 5 years simplifying TensorflowJs for High School students.
https://www.rocksetta.com/tensorflowjs/ 11
My favorite basic layers example is at
https://www.rocksetta.com/tensorflowjs/beginner-keras/20keras-xOr.html 7
I am now getting back to TFJS.
Anyone want to comment on possible changes, or if they are interested in what I do?

Context: @Jason might be interested


Question:
I want to try to explore a little bit the collaboration over the CI/infra.
After evaluating the recent positive impact of having a more transparent and reproducible linting env with a [public Github action] (Actions · tensorflow/tensorflow · GitHub 2) in the repository why we don’t progressively expand a little bit this pilot with self-hosted github runners?
I’ve already talked about this in PR #48421  6 and I’ve mentioned different solutions. As you can see in comments some of the mentioned repo are maintained by Google Cloud team members.
As the dev-infra team is always quite busy having Gtihub Action workflows hosted in the main repository with self-hosted GitHub runners I think it could open the collaboration and partially co-maintainership on the CI infra, minimizing, as possibile, the delta between the internal testing perimeter/env and the public testing space increasing the transparency and community collaboration.
E.g. As you can see in Numpy 1.20 issue/PRs 4 people often don’t understand the full picture of our CI (internal + external).
Minimizing the delta with an expanded public space could help us to build up a more conscious community which is one of the prerequisites for attracting more contributions also in this specific domain.
/cc @thea @Joana @yarri-oss

Context:
We have an internal draft proposal on this but any kind of public feedback is welcome.
Thanks


Question:
Hi, I’m currently using TensorBoard to analyse experiments involving the training and evaluation of time series forecasting models.  I currently create plots of various metrics in TensorBoard (MSE, MAE etc) but am interested in improving the analysis I do through TensorBoard.
Does anyone have any recommendations of:

Useful plugins to add
Extra logging to add
Ways of customising the TensorBoard plots
Anything else that has helped you get the most out of TensorBoard

Thanks in advance!

Context: Hi, have you visited TensorBoard’s official Get started page Get started with TensorBoard  |  TensorFlow 8 ? On the left-hand side - and you may have to expand the browser window to see it - there is a list of different guides. Let us know what you think. In addition, there’s a long tutorial video made by a YouTuber about TensorBoard: TensorFlow Tutorial 17 - Complete TensorBoard Guide - YouTube 3 you may want to check out.


Tensorboard fails to plot model weights for all epochs during training

Question:
I am trying to plot the progression of my model’s weights during training using add_scalar, however, the tensorboard plot only shows the weights for any one epoch.
What I mean by this is, when I load tensorboard, I only see “epoch 0” in the scalars section even if I have run my model for 10 epochs. However, I dont have this issue while plotting histograms in the same code.
My code is as follows:
for epoch in total_epochs:
          train model
          calculate error
          optimizer.step()
          calculate average loss

          for name,weight in model.named_parameters():
              SummaryWriter.add_histogram(name, weight, epoch)
              SummaryWriter.add_scalar(str(name), weight, epoch)

Here 1 is an example of what I mean. I had run the model for 10 epochs, the graph only shows epoch 0 and 1. However, the histogram (not pictured) contains the progression of all 10 epochs.

Context: For custom training loops and TensorBoard, have you tried the method described in  Using TensorBoard with other methods 2 (tf.summary)?


Question:
Hi everyone,
I saw the following tweet last week about splitting Keras into a seperate pip package.
Does this mean that, on my local virtual environment I need to do ‘pip install --upgrade keras’ in order to get the latest tf.keras?
Up until now, keras is updated as part of my:

pip install --upgrade tensorflow"
or
pip install --upgrade tf-nighly"

Thanks!
Fadi

Context:
It is still ok to use:
pip install --upgrade tf-nighly


SIG Build June Meeting: June 8th @ 2pm (delayed a week)

Question:
SIG Build’s next meeting will be on Tuesday, June 8th, at 2pm Pacific time. Find the meeting details here 5. Please feel free to suggest your own agenda items.
Normally the meeting would be on June 1st, but that’s right after Memorial Day, so @perfinion and I decided to move it a week later.

Context: The meeting is happening tomorrow, on Tuesday the 8th! Find the meeting details, how to join, and agenda here 5.


Module ‘tensorflow.compat.v2.__internal__’ has no attribute ‘tf2’

Question:
Hi,  Rizal here.
I am following this topic  , GitHub - AntonMu/TrainYourOwnYOLO: Train a state-of-the-art yolov3 object detector from scratch! 25
but when i do the training, i got the error
2021-06-03 17:39:14.545998: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Using TensorFlow backend.
Traceback (most recent call last):
File “Train_YOLO.py”, line 28, in
import keras.backend as K
File “/home/rizal/.local/lib/python3.8/site-packages/keras/init.py”, line 12, in
from . import initializers
File “/home/rizal/.local/lib/python3.8/site-packages/keras/initializers/init.py”, line 126, in
populate_deserializable_objects()
File “/home/rizal/.local/lib/python3.8/site-packages/keras/initializers/init.py”, line 51, in populate_deserializable_objects
LOCAL.GENERATED_WITH_V2 = tf.internal.tf2.enabled()
AttributeError: module ‘tensorflow.compat.v2.internal’ has no attribute ‘tf2’
Any hint, for me to move forward ? Many thanks in advanded. guys.
Cheers.

Context: @Ahmad_Rizal , This 771 stackoverflow answer may help you.


Question:
Hey all,
Just wondering if anyone has any interest in joining up for the reproducibility challenge this year?

The primary goal of this event is to encourage the publishing and sharing of scientific results that are reliable and reproducible. In support of this, the objective of this challenge is to investigate reproducibility of papers accepted for publication at top conferences by inviting members of the community at large to select a paper, and verify the empirical results and claims in the paper by reproducing the computational experiments, either via a new implementation or using code/data or other information provided by the authors.

The submission deadline is July 15th, 2021, so we don’t have a lot of time but with some effort, we can pull it off. Could be fun to tackle a paper that has shown promise and would be a useful addition to Tensorflow.
Comment below if you think you’ll have a bit of time to spare and what paper you think could be worth reproducing

Context:
This is a nice idea!
Unfortunately I can’t participate at the moment
I’d go even further and publish the model on TFHub when ready!


is it possible to use tensor flow lite with infineon tricore family architecture?

Question: I am currently using the infineon tricore controller. I would like to include the TF-lite model in to this controller. is it possible? how to compile C/C++ code specifically toward the infinion tricore compiler?

Context: /cc @AStevens_Infineon


Code examples for using TensorFlow for Java?

Question:
Hi,
I wasn’t aware of the availability of TensorFlow for Java, which github repository is available here:
GitHub - tensorflow/java: Java bindings for TensorFlow 3
This is great news. Is there any good place to find code examples for training, predictions, model building and the likes?
Thank you,
OD

Context:
We have a small set of example models here: GitHub - tensorflow/java-models: Models in Java 12
TensorFlow Java doesn’t have access to all of the gradients available in TensorFlow as many are implemented in python, so some models can’t be specified entirely in Java yet, but we’re working to build out the set.


Question: Can we have SIG for GDL/3D deep learning? It would be also a good idea to create a tag and resources thread for GDL.

Context:
There was a plan for TF Graphics to start a SIG

      github.com/tensorflow/graphics

      Add Roadmap.md 1

        opened

        Nov 14, 2019

          closed

        May 15, 2020

          bhack

    Is there a roadmap for this TF subprojects.
Seems that It Is very low resource …so It Is interesting if you can share TF plans/expectation on this.

But I don’t know what Is the current status
We have also TF 3D but on GitHub it Is still under Google Reaseach org.

      Google AI Blog

3D Scene Understanding with TensorFlow 3D 8

  Posted by Alireza Fathi, Research Scientist and Rui Huang, AI Resident, Google Research    The growing ubiquity of 3D sensors (e.g., Lidar ,...

/cc @thea


Question:
Currently micro prefers, what in DSP would be referred to as, fixed point representations of all it’s integer tensors. This is very acceptable for situations where the expected dynamic range is  low, i.e. NNs with batch norm, etc. I’m interested in using micro for general audio/other DSP where I’m used to using block floating point.
What are my options here? Any plans to support it/is it supported?
My ultimate goal is to move audio frontend DSP code of an audio pipeline into tf so as to consolidate our DSP(block floating point) and ML(fixed/floating point) frameworks.
Thanks

Context:
I just saw ’ * Dynamic quantized models support’ on TensorFlow Lite Roadmap 2
Is this what I am asking for?


Question: Is anyone using a custom/secondary flatbuffer schema for CustomOptions to ops instead of using FlexBuffers? If so then how has it gone? Thanks

Context:
Is anyone using a custom/secondary flatbuffer schema for CustomOptions to ops instead of using FlexBuffers? If so then how has it gone? Thanks

Looping in @Advait_Jain (TF Lite Micro team)


Hello Everybody - Is this the place for all future discussions?

Question:
Are all meetings happening here in the future?
Yours @ulf1

Context: Yes, we still have monthly  SIG-Addons meetings.


Question: Discuss state-of-the-art ML models both in and out of the TensorFlow model garden.

Context:
Two models I would immediately like to see in Model Garden:

[WideResNets] (very helpful for benchmarking purposes)
Any colorization model

I had worked on a Colorization model 9 in the past months with a collaborator. It gave decent results but we could not get the distributed training part right. We are up for collaborations! Our pipeline is pretty fast so far utilizing the best practices from tf.data and is fully compatible with TPUs.


Keras-cv, keras-nlp, keras applications, models garden

Question:
Can someone give a general overview about the models request we are collecting on Keras-cv?
What kind of relationship we will have between this, models garden, tf.keras.applications namespace and “marginally” TFHUB?
I would be really nice to disambiguate a little bit this topic to avoid duplication, fragmentation and confusion about the contribution path in the TF ecosystem and to optimize the external contributors resources.
We have already have some historically pinned tickets about models garden community requests and help wanted requests at:

      github.com/tensorflow/models

      📄 Community requests: New paper implementations 2

        opened

        Jun 6, 2020

          jaeyounkim

          type:support

          models:official

    This issue contains **all open requests for paper implementations requested by t…he community**.

We cannot guarantee that we can fulfill community requests for specific paper implementations.

If you'd like to contribute, **please add a comment to the relevant GitHub issue to express your interest in providing your paper implementation**.

Awesome external contributors will be nominated for [Google Open Source Peer Bonus](https://opensource.google/docs/growing/peer-bonus/).

Please also see our [contribution guidelines](https://github.com/tensorflow/models/wiki/Research-paper-code-contribution) and [paper selection criteria](https://github.com/tensorflow/models/wiki/Research-paper-code-contribution#model-selection).

## Computer Vision

| Paper | Conference | GitHub issue | Note |
--------|------------|--------------|------|
| ResNeXt: [Aggregated Residual Transformations for Deep Neural Networks](https://arxiv.org/abs/1611.05431) | CVPR 2017 | #6752 | |
| DenseNet: [Densely Connected Convolutional Networks](https://arxiv.org/abs/1608.06993) | CVPR 2017 | #8278 | |
| [Density estimation using Real NVP](https://arxiv.org/abs/1605.08803) | ICLR 2017 | #7848 | Need to migrate [TF 1 code](https://github.com/tensorflow/models/tree/master/research/real_nvp) to TF 2 |
| [Spatiotemporal Contrastive Video Representation Learning](https://arxiv.org/abs/2008.03800) |  CVPR 2021 | #9993 |  In progress (Internally) |

      github.com/tensorflow/models

      [Help wanted] Research paper implementations (Project Tracker) 1

        opened

        Jun 21, 2020

          jaeyounkim

          help wanted:paper implementation

    # Help wanted: Research paper code and models

This issue contains a list of t…he research papers we want to implement in TensorFlow 2 with help from the community.

If you'd like to contribute, please **add a comment to the relevant GitHub issue** or **create a new issue** to express your interest in providing your paper implementation.

Awesome external contributors will be nominated for [Google Open Source Peer Bonus](https://opensource.google/docs/growing/peer-bonus/).

Please also see our [contribution guidelines](https://github.com/tensorflow/models/wiki/Research-paper-code-contribution) and [paper selection criteria](https://github.com/tensorflow/models/wiki/Research-paper-code-contribution#model-selection).

## Computer Vision

| Paper | GitHub issue | Status |
|-------|--------------|--------|
| FCOS: Fully Convolutional One-Stage Object Detection | #10275 |  In progress |
| DarkPose: [Distribution Aware Coordinate Representation for Human Pose Estimation](https://arxiv.org/abs/1910.06278) | #8713 | In progress |
| MoCo: [Momentum Contrast for Unsupervised Visual Representation Learning](https://arxiv.org/abs/1911.05722) | #8708 | Need contribution |
| YOLOv4 [Optimal Speed and Accuracy of Object Detection](https://arxiv.org/abs/2004.10934) | N/A | [In progress](https://github.com/tensorflow/models/tree/master/official/vision/beta/projects/yolo) |

## Natural Language Processing

| Paper | GitHub issue | Status |
|-------|--------------|--------|
| RoBERTa: [A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) | #8704 | Need contribution |
| RoFormer: Enhanced Transformer with Rotary Position Embedding | N/A | In progress |
| Longformer: The Long-Document Transformer | N/A | In progress |
| BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension | N/A | In progress |

### Benchmark datasets

| Dataset | GitHub issue(s) | Status |
|----------|------------------|--------|

## Speech Recognition

| Paper | Conference | GitHub issue | Status |
|-------|------------|--------------|--------|
| Deep Speech 2: [End-to-End Speech Recognition in English and Mandarin](https://arxiv.org/abs/1512.02595) | ICML 2016 | #8702 | In progress |

Useful for the context
See also other community members comments like @sebastian-sz :

      github.com/keras-team/keras-cv

      ResNet-RS block/layer 1

        opened

        Jan 12, 2022

          LukeWood

Or our thread at:

      github.com/keras-team/keras

      Updating the ResNet-* weights

        opened

        Dec 12, 2021

          sayakpaul

          type:feature

          Contributions welcome

    If you open a GitHub issue, here is our policy:

It must be a bug, a feature r…equest, or a significant problem with the documentation (for small docs fixes please send a PR instead).
The form below must be filled out.

**Here's why we have that policy:**.

Keras developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

**System information**.

TensorFlow version (you are using): 2.7.0
Are you willing to contribute it (Yes/No) : Currently no

**Describe the feature and the current behavior/state**.

ResNets are arguably one of the most influential architectures in deep learning. Today, they are used in different capacities. For example, sometimes they act as strong baselines, sometimes they are used as backbones. Since their inception, their performance on ImageNet-1k, in particular, has improved quite a lot. I think it's time the ResNets under `tf.keras.applications` were updated to facilitate these changes.

**Will this change the current api? How?**

ResNet-RS (https://arxiv.org/abs/2103.07579) introduces slight architectural changes to the vanilla ResNet architecture (https://arxiv.org/abs/1512.03385). So, yes, there will be changes to the current implementation of ResNets (among other things) we have under `tf.keras.applications`. We could call it `tf.keras.applications.ResNet50RS`, for example. Following summarizes the performance benefits that ResNet-RS introduces to the final ImageNet-1k performance (measured on the `val` set):

![image](https://user-images.githubusercontent.com/22957388/145714574-b057a95e-4d6c-48ba-b7ea-67e953e83f29.png)

Source

**Who will benefit from this feature?**

Keras users that use ResNets from `tf.keras.applications` for building downstream applications.

**[Contributing](https://github.com/keras-team/keras/blob/master/CONTRIBUTING.md)**

- Do you want to contribute a PR? (yes/no): Currently no
- If yes, please read [this page](https://github.com/keras-team/keras/blob/master/CONTRIBUTING.md) for instructions
- Briefly describe your candidate solution(if contributing):

/cc  @thea @yarri-oss @lgusm @Luke_Wood @Jaehong_Kim @Scott_Zhu

Context:
Thanks Bhack for the question.
We will have some readme/contribution guild available on the keras-cv github project to provide more details about what’s the difference between keras-cv/model garden/keras.applications/tf-hub.


Question: Can Tensorflow\ Keras be used for a synchronous \ holistic modeling off NN architecture\ structure, types of neurons, weights etc? Are there examples to show that?

Context: Do you have any non Keras example/reference? Just to understand a little bit the topic.


[Research ] MLP-Mixer: An all-MLP Architecture for Vision

Question:
MLP-Mixer: An all-MLP Architecture for Vision (Tolstikhin et al., 2021) 16 (Google)

Convolutional Neural Networks (CNNs) are the go-to model for computer vision.
Recently, attention-based networks, such as the Vision Transformer, have also
become popular. In this paper we show that while convolutions and attention are
both sufficient for good performance, neither of them are necessary. We present
MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs).
MLP-Mixer contains two types of layers: one with MLPs applied independently to
image patches (i.e. “mixing” the per-location features), and one with MLPs applied
across patches (i.e. “mixing” spatial information). When trained on large datasets,
or with modern regularization schemes, MLP-Mixer attains competitive scores on
image classification benchmarks, with pre-training and inference cost comparable
to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.

image1218×851 233 KB

… our architecture can be seen as a very special CNN, which uses 1×1 convolutions
for channel mixing, and single-channel depth-wise convolutions of a full receptive field and parameter sharing for token mixing. However, the converse is not true as typical CNNs are not special cases of Mixer. Furthermore, a convolution is more complex than the plain matrix multiplication in MLPs as it requires an additional costly reduction to matrix multiplication and/or specialized implementation.
Despite its simplicity, Mixer attains competitive results. When pre-trained on large datasets (i.e., ∼100M images), it reaches near state-of-the-art performance, previously claimed by CNNs and Transformers, in terms of the accuracy/cost trade-off. This includes 87.94% top-1 validation accuracy on ILSVRC2012 “ImageNet” [13]. When pre-trained on data of more modest scale (i.e., ∼1– 10M images), coupled with modern regularization techniques [48, 53], Mixer also achieves strong performance. However, similar to ViT, it falls slightly short of specialized CNN architectures.

image1118×724 352 KB

We describe a very simple architecture for vision. Our experiments demonstrate that it is as good as existing state-of-the-art methods in terms of the trade-off between accuracy and computational resources required for training and inference. We believe these results open many questions. On the practical side, it may be useful to study the features learned by the model and identify the main differences (if any) from those learned by CNNs and Transformers. On the theoretical side, we would like to understand the inductive biases hidden in these various features and eventually their role in generalization. Most of all, we hope that our results spark further research, beyond the realms of established models based on convolutions and self-attention. It would be particularly interesting to see whether such a design works in NLP or other domains.

@Sayak_Paul’s implementation:

    MLP-Mixer with CIFAR-10 Show and Tell

    Here’s my implementation of MLP-Mixer, the all MLP architecture for computer vision without any use of convs and self-attention:

Here’s what is included:

Distributed training with mixed-precision.
Visualization of the token-mixing MLP weights.
A TensorBoard callback to keep track of the learned linear projections of the image patches.

Results are quite competitive with room for improvements for interpretability.

      github.com

sayakpaul/MLP-Mixer-CIFAR10 6

  Implements MLP-Mixer (https://arxiv.org/abs/2105.01601) with the CIFAR-10 dataset.

Context:
I think that we need to better promote our TF models in paperswithcode /cc @Joana.
In this specific case you can see the official Google reference  implementation in JAX and all the others alternative implementations:

      paperswithcode.com

Papers with Code - MLP-Mixer: An all-MLP Architecture for Vision 15

  #11 best model for Image Classification on ImageNet ReaL (Accuracy metric)


ANN models and the identifiability problem

Question:
Dear community,
In general, ANN models are non-identified. But, we attempt to address the identifiability problem by imposing some constraints. How can I impose these constraints using tf.keras.constraints?
Someone already did this?
Regards.

Context:
If you are looking for how to just handle some constrains optimizzation you could take a look at:
Google AI Blog: Setting Fairness Goals with the TensorFlow Constrained Optimization Library 1


Question:
Hi Everyone,
I am new to Tensorflow (beginner). I am researching solutions for a project i’m working on and was wondering if someone could point me in the right direction. I checked models at modelzoo and it was a bit overwhelming and so far did not find what i was looking for. It seems like a rather common use-case so thought there might already be something built with tensorflow for this, but maybe another tool is better suited?
Basically I want to match incoming photos taken by client with a database of photos and info.
Kind regards,
-M

Context:
You can start to play with something like:

    Near-duplicate image search Show and Tell

    New example on building a near-duplication image search utility. Comprises of an image classifier, Bit LSH (Locality Sensitive Hashing), random projection, and TensorRT optimized inference to drastically reduce the query time. LSH and random projection have been shown with from-scratch implementations for readers to better understand them.

Check also this thread:

    Tutorials/materials for fast image retrieval tasks General Discussion

    Hi all.
Looking for resources/materials that build on TensorFlow/Keras (preferably) for doing scalable image similarity searches.


Which one are the best Densnet or Resnet?

Question: Can someone explain which one are the best algorithm ? I know but I want to learn from a person who may best experience in above. Thanks

Context: On what kind of public dataset?


Question:
Hey Community, I hope you’re doing great.
I’m working on binary classification using structured data, and my model has gave a great validation recall result(around 80%) but low validation accuracy(around 40%)!
and i like to improve the validation accuracy even if that will slightly decrease the validation recall? any suggestions please.
Thank you so much!

Context:
Plotting the “precision vs. recall” curve will show you the performance you can reach just by changing the threshold level. Have a look at this tutorial:

      TensorFlow

Classification on imbalanced data  |  TensorFlow Core 3


Changing augmentation parameters in TFOD API

Question:
I am aware of the preprocessing proto that is used in the models repo:

      github.com

    tensorflow/models/blob/master/research/object_detection/protos/preprocessor.proto 3

    syntax = "proto2";

package object_detection.protos;

// Message for defining a preprocessing operation on input data.
// See: //third_party/tensorflow_models/object_detection/core/preprocessor.py
// Next ID: 41
message PreprocessingStep {
  oneof preprocessing_step {
    NormalizeImage normalize_image = 1;
    RandomHorizontalFlip random_horizontal_flip = 2;
    RandomPixelValueScale random_pixel_value_scale = 3;
    RandomImageScale random_image_scale = 4;
    RandomRGBtoGray random_rgb_to_gray = 5;
    RandomAdjustBrightness random_adjust_brightness = 6;
    RandomAdjustContrast random_adjust_contrast = 7;
    RandomAdjustHue random_adjust_hue = 8;
    RandomAdjustSaturation random_adjust_saturation = 9;
    RandomDistortColor random_distort_color = 10;
    RandomJitterBoxes random_jitter_boxes = 11;

  This file has been truncated. show original

My question is how one configures their augmentation pipeline when using the TFOD API. Consider this configuration file. It has a field for augmentation:
train_config: {
  ...
  data_augmentation_options {
    random_horizontal_flip {
    }
  }

If I wanted to expand the set of augmentation transformations here what should I do?
@Laurence_Moroney @khanhlvg any pointers?

Context:
Interesting.
This looks like they’ve re-encoded something like keras’s model.get_config, as a proto.
To change the data-augmentation, you edit that data_augmentation_options list.
The .proto files define what’s allowed. The definition of TrainConfig is here:

      github.com

    tensorflow/models/blob/aa3e639f80c2967504310b0f578f0f00063a8aff/research/object_detection/protos/train.proto#L25

// Message for configuring DetectionModel training jobs (train.py).
          // Next id: 31
          message TrainConfig {
            // Effective batch size to use for training.
            // For TPU (or sync SGD jobs), the batch size per core (or GPU) is going to be
            // `batch_size` / number of cores (or `batch_size` / number of GPUs).
            optional uint32 batch_size = 1 [default=32];

  // Data augmentation options.
            repeated PreprocessingStep data_augmentation_options = 2;

  // Whether to synchronize replicas during training.
            optional bool sync_replicas = 3 [default=false];

  // How frequently to keep checkpoints.
            optional float keep_checkpoint_every_n_hours = 4 [default=10000.0];

  // Optimizer used to train the DetectionModel.
            optional Optimizer optimizer = 5;

data_augmentation_options is a repeated PreprocessingStep.
A PreprocessingStep is one of the items from that list. The parameters of each and their default values are defined in preprocessor.proto
If you want to add a RandomScale step:
train_config: {
  ...
  data_augmentation_options {
    random_horizontal_flip {
    }
   random_image_scale {
       min_scale_ratio: 0.9
       max_scale_ratio: 1.1
    }
  }
}

That format is “proto-text” (.PBTXT), you can check your syntax with:
from google.protobuf import text_format

train_config = TrainConfig()
train_config = text_format.Parse(
        r"""
        train_config: {
          ...
          data_augmentation_options {
            random_horizontal_flip {
            }
           random_image_scale {
               min_scale_ratio: 0.9
               max_scale_ratio: 1.1
            }
          }
        }
        """, train_config)
print(train_config)


Collective Intelligence for Deep Learning

Question:
A very interesting survey by Google Brain Tokyo:
https://arxiv.org/abs/2111.14377 15

Context:
In the case you are looking for a gentle introduction to the “emergence” I recommend you this 2014 blog post by David Pines:

      Medium – 13 Nov 14

Emergence: A unifying theme for 21st century science 4

  By David Pines, Co-Founder in Residence, Santa Fe Institute

    Reading time: 13 min read


Prepare .wav file for yamnet.tflite model

Question:
Hi developers?
How to prepare  .wav and  .amr file for yamnet.tflite model in kotlin or java. I have checked the example project on Github but it has only real-time classification using the mic, but I need to know how to prepare the wav and amr file for this model. thanks

Context:
Hello sir, I hope you’re well, I found the solution for yamnet model, and write the article on Medium,

      Medium – 19 Oct 21

Prepare .wav or .amr files for yamnet.tflite model Android 5

  After a lot of research, I realized there is no any easiest way to prepare a local .wav or .amr file for yamnet.tflite model, as we know…

    Reading time: 1 min read

Please check it out and give me suggestions to improve it.


Question:
Hi everyone,
I’m doing some research on Unified Memory Management on Multi-GPUs system and trying to compare the performance with explicit copy on some real ML workloads.
The benefits from Unified Memory are

Allow memory oversubscription
Improve programmability, programmers don’t need to worry about data placement and movement

I found there’s a switch per_process_gpu_memory_fraction to turn on Unified Memory in tensorflow. For distributed training on multi GPUs, I used tf.distribute.MirroredStrategy API. But from profiling result, it seems that tensorflow just leverage Unified Memory to overcome memory oversubscription, there are still explicit memory copies between GPU and CPU, or GPU and GPU.
I’m wondering if there’s a way to train on multi GPUs and fully explore the power of Unified Memory, like letting memory system manage the data, in tensorflow.
System information

TensorFlow version (you are using): 2.4
CUDA version: 11.0
cudnn version: 8.0

Thanks

Context:
Have you tried with these envs on TF 2.7:

      github.com/tensorflow/tensorflow

      [PJRT] Allow GPU memory oversubscription when unified memory is enabled. 6

        committed

        Jul 9, 2021

          tensorflower-gardener

          +5
          -2

    With this CL, we can enable GPU memory oversubscription via env flags.
For examp…le, `TF_FORCE_UNIFIED_MEMORY=1 XLA_PYTHON_CLIENT_MEM_FRACTION=8.0` provides 8x the GPU memory to the program. The 'extra' memory is physically located on the other GPU devices and the host's RAM, with swapping done transparently by CUDA.

PiperOrigin-RevId: 383819164
Change-Id: Id139d3184d3a62983c1e86bf95ca4078a08db4f4


[Google Research ] Self-supervised learning - Compressed SimCLR / BYOL with Conditional Entropy Bottleneck (with TensorFlow code)

Question:
New self-supervised methods—Compressed SimCLR and Compressed BYOL with Conditional Entropy Bottleneck—for learning effective and robust visual representations, which enable learning visual classifiers with limited data.

arXiv: Compressive Visual Representations 5 (Lee et al., 2021) (Google Research)

Learning effective visual representations that generalize well without human supervision is a fundamental problem in order to apply Machine Learning to a wide variety of tasks. Recently, two families of self-supervised methods, contrastive learning and latent bootstrapping, exemplified by SimCLR and BYOL respectively, have made significant progress. In this work, we hypothesize that adding explicit information compression to these algorithms yields better and more robust representations. We verify this by developing SimCLR and BYOL formulations compatible with the Conditional Entropy Bottleneck (CEB) objective, allowing us to both measure and control the amount of compression in the learned representation, and observe their impact on downstream tasks. Furthermore, we explore the relationship between Lipschitz continuity and compression, showing a tractable lower bound on the Lipschitz constant of the encoders we learn. As Lipschitz continuity is closely related to robustness, this provides a new explanation for why compressed models are more robust. Our experiments confirm that adding compression to SimCLR and BYOL significantly improves linear evaluation accuracies and model robustness across a wide range of domain shifts. In particular, the compressed version of BYOL achieves 76.0% Top-1 linear evaluation accuracy on ImageNet with ResNet-50, and 78.8% with ResNet-50 2x.1

Recent contrastive approaches to self-supervised visual representation learning aim to learn representations that maximally capture the mutual information between two transformed views of an image… The primary idea of these approaches is that this mutual information corresponds to a general shared context that is invariant to various transformations of the input, and it is assumed that such invariant features will be effective for various downstream higher-level tasks. However, although existing contrastive approaches maximize mutual information between augmented views of the same input, they do not necessarily compress away the irrelevant information from these views… retaining irrelevant information often leads to less stable representations and to failures in robustness and generalization, hampering the efficacy of the learned representations. An alternative state-of-the-art self-supervised learning approach is BYOL [30], which uses a slow-moving average network to learn consistent, view-invariant representations of the inputs. However, it also does not explicitly capture relevant compression in its objective.
In this work, we modify SimCLR [12], a state-of-the-art contrastive representation method, by adding information compression using the Conditional Entropy Bottleneck (CEB) [27]. Similarly, we show how BYOL [30] representations can also be compressed using CEB. By using CEB we are able to measure and control the amount of information compression in the learned representation [26], and observe its impact on downstream tasks. We empirically demonstrate that our compressive variants of SimCLR and BYOL, which we name C-SimCLR and C-BYOL, significantly improve accuracy and robustness to domain shifts across a number of scenarios.

Code: GitHub - google-research/compressive-visual-representations: Tensorflow 2 implementations of the C-SimCLR and C-BYOL self-supervised visual representation methods from "Compressive Visual Representations" (NeurIPS 2021) 2

C-SimCLR866×259 129 KB

C-BYOL866×337 182 KB

Related work:

SimCLR: A Simple Framework for Contrastive Learning of Visual Representations (Chen et al., 2020) (Google Research, Brain Team)

SimCLRv2: Big Self-Supervised Models are Strong Semi-Supervised Learners (Chen et al., 2020) (Google Research, Brain Team)
GitHub with TF 2 implementation

BYOL: Bootstrap your own latent: A new approach to self-supervised Learning (Grill et al., 2020) (DeepMind/Imperial College)

Context:
Interested in learning about self-supervised methods? Here are some resources:

Google AI Blog: Advancing Self-Supervised and Semi-Supervised Learning with SimCLR
Google AI Blog: Extending Contrastive Learning to the Supervised Setting

Some code examples and other posts made by the ML community members:

Keras: Self-supervised contrastive learning with SimSiam (by @Sayak_Paul)

GitHub - sayakpaul/SimCLR-in-TensorFlow-2: (Minimally) implements SimCLR (https://arxiv.org/abs/2002.05709) in TensorFlow 2. (by @Sayak_Paul)

GitHub - ayulockin/SwAV-TF: TensorFlow implementation of "Unsupervised Learning of Visual Features by Contrasting Cluster Assignments". (by ayulockin and @Sayak_Paul)

GitHub - sayakpaul/SimSiam-TF: Minimal implementation of SimSiam (https://arxiv.org/abs/2011.10566) in TensorFlow 2. (by @Sayak_Paul)
Lilian Weng’s blog post: Self-Supervised Representation Learning (2019)
Lilian Weng’s blog post: Contrastive Representation Learning (2021)


Question:
It could be really nice if we could launch an experiment/initiative with the Model Garden, Federated teams and our community on training our first model garden model with our federated tools.
We have some recent interesting experiment with other frameworks.

      learning@home

Train vast neural networks together

  A library to train large neural networks across the internet. Imagine training one huge transformer on thousands of computers from universities, companies, and volunteers.

https://arxiv.org/abs/2106.10207

      huggingface.co

neuropark/sahajBERT · Hugging Face

Context: Hello, TensorFlow already does and you can use KubelFlow, ML overlay from Kubernetes…


Where can I find sysmetric TFlite quantization model?

Question:
I would like to find some symmetric 8 bit quantization models to deploy on hardware. However, every model I found in model zoo are asymmetric. Hosted models  |  TensorFlow Lite

      TensorFlow

TensorFlow Lite 8-bit quantization specification

Here mentioned that “In the past our quantization tooling used per-tensor, asymmetric, uint8 quantization. New tooling, reference kernels, and optimized kernels for 8-bit quantization will use this spec.”
Do you mean that symmetric quantizated tflite models are still under development? Or I could find some symmetric quantization tflite model elsewhere?

Context:
We have a tracking ticket at:

      github.com/tensorflow/tensorflow

      Allow symmetric TFLite quantization (no zero point/scale only)  2

        opened

        Sep 10, 2020

          bwang1991

          stat:awaiting tensorflower

          type:feature

          comp:lite

    As far as I know, TFLite's quantization forces activations to have both scales a…nd zero points. However, for some networks, symmetric quantization (no zero point) does not cause a significant loss in accuracy. It is therefore sufficient to use scale only. Please add support for symmetric quantization.


From keras.engine. topology import network

Question:
What are the possible ways to install topology for Keras. Because the following import command gives error.
Command:
from keras.engine.topology import network
Error:
ModuleNotFoundError: No module named ‘keras.engine.topology’

Context:
This might depend on the Keras version you are using. Look into if the keras.engine.topology has depricated.
You can force install an earlier version by:
pip install 'keras==2.1.6' --force-reinstall
Where 2.1.6 is a suitable example. You may  try
import tensorflow.python.keras.engine
But you will not be able to import topology from tensorflow.python.keras.engine .
Please refer to the answers in similar issue1 12, issue2 9.
Thanks!


How can I measure the complexity of the model?

Question:
I need to measure time and memory complexity for a keras model or captioning model using keras … how can I start ?
Thanks

Context:
Check:

    How to find out keras model memory size? General Discussion

    Hello!
I am doing a school work and I need to find out keras model memory size so I could compare different models. It is supposed to be composed of weights/parameters and model itself. It was given that model.summary() should contain all the information. From there I see that layer info and output shapes with the number of parameters.
I understand parameters as they are just a numbers and so, number of parameters * 4B most likely will give how much room parameters take. But I know that more i…


The H5 model I trained can be used normally on the computer after being converted to tflite model, but it can’t be used on Android devices. What’s the problem

Question: I trained the H5 model through the computer and passed tf.lite.tfliteconverter.from_ keras_ Mode to convert it into tflite model, but it seems that it can’t be used on Android devices. The reasoning result obtained through interpreter.run is Nan. Where did I make a configuration error?

Context:
Hi @jun_yin
I am a little bit confused
Can you use the tflite model with the python api?


Question: My dataset is a combination of time series and non-time series data. what model can I use to train my machine using both types of data?

Context: @Md_Samiul_Basir, Model selection is purely based business problem. Is is possible to share sample data to understand more?


Where should I place dropout layer in the network

Question:
how and where can I add drop out layer in the following code:
from tensorflow.keras.layers import Conv2D, BatchNormalization, Activation, MaxPool2D, Conv2DTranspose, Concatenate, Input
from tensorflow.keras.models import Model
from tensorflow.keras.applications import ResNet50
def conv_block(input, num_filters):
x = Conv2D(num_filters, 3, padding=“same”)(input)
x = BatchNormalization()(x)
x = Activation(“relu”)(x)
x = Conv2D(num_filters, 3, padding="same")(x)
x = BatchNormalization()(x)
x = Activation("relu")(x)

return x

def decoder_block(input, skip_features, num_filters):
x = Conv2DTranspose(num_filters, (2, 2), strides=2, padding=“same”)(input)
x = Concatenate()([x, skip_features])
x = conv_block(x, num_filters)
return x
def resnet50_unet(input_shape):
“”" Input “”"
inputs = Input(input_shape)
""" Pre-trained ResNet50 Model """
resnet50 = ResNet50(include_top=False, weights=None, input_tensor=inputs)

""" Encoder """
s1 = resnet50.layers[0].output          ## (512 x 512)
s2 = resnet50.get_layer("conv1_relu").output        ## (256 x 256)
s3 = resnet50.get_layer("conv2_block3_out").output  ## (128 x 128)
s4 = resnet50.get_layer("conv3_block4_out").output  ## (32 x 32)

""" Bridge """
b1 = resnet50.get_layer("conv4_block6_out").output  ## (32 x 32)

""" Decoder """
d1 = decoder_block(b1, s4, 256)                     ## (64 x 64)
d2 = decoder_block(d1, s3, 128)                     ## (128 x 128)
d3 = decoder_block(d2, s2, 64)                     ## (256 x 256)
d4 = decoder_block(d3, s1, 32)                      ## (512 x 512)

""" Output """
outputs = Conv2D(1, 1, padding="same", activation="sigmoid")(d4)

model = Model(inputs, outputs, name="ResNet50_U-Net")
return model

if name == “main”:
input_shape = (256, 256, 3)
model = resnet50_unet(input_shape)
#model.summary()

Context:
@Aleena_Suhail,
Can you take a look at this thread 7 which can throw you some insights on the same?


How to improve accuracy of a CNN_LSTM binary classifier in TF 2.4

Question:
I am trying to build a CNN LSTM classifier for 1d sequential data.Input is of length 20 and contains 4 features.
I have trained the model and saved it. However I am unable to get good performance in both training as well as test data:-
Below is my code for the tensorflow model.
model = tf.keras.Sequential()
model.add(tf.keras.layers.Conv1D(filters=128, kernel_size=8, padding = 'same', activation='relu', input_shape = (20,4)))
model.add(tf.keras.layers.Conv1D(filters=128, kernel_size=5, padding = 'same', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01)))
model.add(tf.keras.layers.Conv1D(filters=128, kernel_size=3, padding = 'same', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01)))
model.add(tf.keras.layers.MaxPooling1D(pool_size=2))
model.add(tf.keras.layers.LSTM(units = 128))
model.add(tf.keras.layers.Dense(units = 1, activation = 'sigmoid'))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics = 'accuracy')
model.build()
model.summary()
history = model.fit(X_tf, y_tf, epochs=60, batch_size=256, validation_data = (X_tf_,y_tf_))

Here are the logs that I am getting while training.

Epoch 5/60 19739/19739 [==============================] - 1212s 61ms/step - loss: 0.5858 - accuracy: 0.7055 - val_loss: 0.5854 - val_accuracy: 0.7062

I need help in how can I further improve the performance.What are the various techniques that I can apply to sequential data?
My training dataset has 4.8 million rows and test set has 1.2 million rows.

Context:
You can make the model bigger: add more LSTM layers, increase the number of units in the layers, make them bidirectional, add dense layers with activations after the last LSTM or experiment with other architectures.
Other way is to change the number of epochs, batch size and learning rate and see how it affects the results.
If nothing helps, check for class imbalance and how both classes are distributed between the train and validation sets. Apply some basic techniques for imbalances data like using sample weights and generating synthetic data for underrepresented class.
Add more features, if it is possible, or generate new features from existing ones.


[Research ] Weather forecasting for up to 12 hours with MetNet-2 (Google Research)

Question:
Meteorological Neural Network 2 (MetNet-2) - a new probabilistic weather model featuring a forecasting range of up to 12 hours of lead time at a frequency of 2 minutes.
Blog post: Google AI Blog: MetNet-2: Deep Learning for 12-Hour Precipitation Forecasting 7

Within weather forecasting, deep learning techniques have shown particular promise for nowcasting 1 — i.e., predicting weather up to 2-6 hours ahead. Previous work has focused on using direct neural network models for weather data 1, extending neural forecasts from 0 to 8 hours with the MetNet architecture, generating continuations of radar data for up to 90 minutes ahead, and interpreting the weather information learned by these neural networks. Still, there is an opportunity for deep learning to extend improvements to longer-range forecasts.
To that end, in “Skillful Twelve Hour Precipitation Forecasts Using Large Context Neural Networks”, we push the forecasting boundaries of our neural precipitation model to 12 hour predictions while keeping a spatial resolution of 1 km and a time resolution of 2 minutes. By quadrupling the input context, adopting a richer weather input state, and extending the architecture to capture longer-range spatial dependencies, MetNet-2 substantially improves on the performance of its predecessor, MetNet. Compared to physics-based models, MetNet-2 outperforms the state-of-the-art HREF ensemble model for weather forecasts up to 12 hours ahead.

Interpreting What MetNet-2 Learns About Weather
Because MetNet-2 does not use hand-crafted physical equations, its performance inspires a natural question: What kind of physical relations about the weather does it learn from the data during training? Using advanced interpretability tools 1, we further trace the impact of various input features on MetNet-2’s performance at different forecast timelines. Perhaps the most surprising finding is that MetNet-2 appears to emulate the physics described by Quasi-Geostrophic Theory, which is used as an effective approximation of large-scale weather phenomena. MetNet-2 was able to pick up on changes in the atmospheric forces, at the scale of a typical high- or low-pressure system (i.e., the synoptic scale), that bring about favorable conditions for precipitation, a key tenet of the theory.
Conclusion
MetNet-2 represents a step toward enabling a new modeling paradigm for weather forecasting that does not rely on hand-coding the physics of weather phenomena, but rather embraces end-to-end learning from observations to weather targets and parallel forecasting on low-precision hardware. Yet many challenges remain on the path to fully achieving this goal, including incorporating more raw data about the atmosphere directly (rather than using the pre-processed starting state from physical models), broadening the set of weather phenomena, increasing the lead time horizon to days and weeks, and widening the geographic coverage beyond the United States.

MetNet-2 architecture1448×575 208 KB

(MetNet-2 architecture - paper 5)

Also, check out:

    [Research ] Nowcasting the Next Hour of Rain (by DeepMind) Research & Models

    GitHub: https://github.com/deepmind/deepmind-research/tree/master/nowcasting
Colab: Google Colab
Paper: Skilful precipitation nowcasting using deep generative models of radar | Nature

Recently introduced deep learning methods use radar to directly predict future rain rates, free of physical constraints5,6. While they accurately predict low-intensity rainfall, their operational utility is limited because their lack of constraints produces blurry nowcasts at longer lead times, yielding poor …


Question:
def expend_as(tensor, rep):
return layers.Lambda(lambda x, repnum: K.repeat_elements(x, repnum, axis=3),
arguments={‘repnum’: rep})(tensor)
def double_conv_layer(x, filter_size, size, dropout, batch_norm=False):
axis = 3
conv = layers.Conv2D(size, (filter_size, filter_size), padding='same')(x)
if batch_norm is True:
    conv = layers.BatchNormalization(axis=axis)(conv)
conv = layers.Activation('relu')(conv)
conv = layers.Conv2D(size, (filter_size, filter_size), padding='same')(conv)
if batch_norm is True:
    conv = layers.BatchNormalization(axis=axis)(conv)
conv = layers.Activation('relu')(conv)
if dropout > 0:
    conv = layers.Dropout(dropout)(conv)

shortcut = layers.Conv2D(size, kernel_size=(1, 1), padding='same')(x)
if batch_norm is True:
    shortcut = layers.BatchNormalization(axis=axis)(shortcut)

res_path = layers.add([shortcut, conv])
return res_path

def gating_signal(input, out_size, batch_norm=False):
“”"
resize the down layer feature map into the same dimension as the up layer feature map
using 1x1 conv
:param input:   down-dim feature map
:param out_size:output channel number
:return: the gating feature map with the same dimension of the up layer feature map
“”"
x = layers.Conv2D(out_size, (1, 1), padding=‘same’)(input)
if batch_norm:
x = layers.BatchNormalization()(x)
x = layers.Activation(‘relu’)(x)
return x
def attention_block(x, gating, inter_shape):
shape_x = K.int_shape(x)
shape_g = K.int_shape(gating)
theta_x = layers.Conv2D(inter_shape, (2, 2), strides=(2, 2), padding='same')(x)  # 16
shape_theta_x = K.int_shape(theta_x)

phi_g = layers.Conv2D(inter_shape, (1, 1), padding='same')(gating)
upsample_g = layers.Conv2DTranspose(inter_shape, (3, 3),
                             strides=(shape_theta_x[1] // shape_g[1], shape_theta_x[2] // shape_g[2]),
                             padding='same')(phi_g)  # 16

concat_xg = layers.add([upsample_g, theta_x])
act_xg = layers.Activation('relu')(concat_xg)
psi = layers.Conv2D(1, (1, 1), padding='same')(act_xg)
sigmoid_xg = layers.Activation('sigmoid')(psi)
shape_sigmoid = K.int_shape(sigmoid_xg)
upsample_psi = layers.UpSampling2D(size=(shape_x[1] // shape_sigmoid[1], shape_x[2] // shape_sigmoid[2]))(sigmoid_xg)  # 32

upsample_psi = expend_as(upsample_psi, shape_x[3])

y = layers.multiply([upsample_psi, x])

result = layers.Conv2D(shape_x[3], (1, 1), padding='same')(y)
result_bn = layers.BatchNormalization()(result)
return result_bn

def Attention_ResUNet(input_shape, NUM_CLASSES=1, dropout_rate=0.0, batch_norm=True):
FILTER_NUM = 64 # number of basic filters for the first layer
FILTER_SIZE = 3 # size of the convolutional filter
UP_SAMP_SIZE = 2
# input data
# dimension of the image depth
inputs = layers.Input((512, 512, 3), dtype=tf.float32)
axis = 3
# Downsampling layers
# DownRes 1, double residual convolution + pooling
conv_512 = double_conv_layer(inputs, 3, 64, dropout_rate, batch_norm)
pool_256 = layers.MaxPooling2D(pool_size=(2,2))(conv_512)
# DownRes 2
conv_256 = double_conv_layer(pool_256, 3, 2*64, dropout_rate, batch_norm)
pool_128 = layers.MaxPooling2D(pool_size=(2,2))(conv_256)
# DownRes 3
conv_128 = double_conv_layer(pool_128, 3, 4*64, dropout_rate, batch_norm)
pool_64 = layers.MaxPooling2D(pool_size=(2,2))(conv_128)
# DownRes 4
conv_64 = double_conv_layer(pool_64, 3, 8*64, dropout_rate, batch_norm)
pool_32 = layers.MaxPooling2D(pool_size=(2,2))(conv_64)
# DownRes 5, convolution only
conv_32 = double_conv_layer(pool_32, 3, 16*64, dropout_rate, batch_norm)

# Upsampling layers
# UpRes 6, attention gated concatenation + upsampling + double residual convolution
gating_64 = gating_signal(conv_32, 8*64, batch_norm)
att_64 = attention_block(conv_64, gating_64, 8*64)
up_64 = layers.UpSampling2D(size=(2, 2), data_format="channels_last")(conv_32)
up_64 = layers.concatenate([up_64, att_64], axis=axis)
up_conv_64 = double_conv_layer(up_64, 3, 8*64, dropout_rate, batch_norm)
# UpRes 7
gating_128 = gating_signal(up_conv_64, 4*64, batch_norm)
att_128 = attention_block(conv_128, gating_128, 4*64)
up_128 = layers.UpSampling2D(size=(2, 2), data_format="channels_last")(up_conv_64)
up_128 = layers.concatenate([up_128, att_128], axis=axis)
up_conv_128 = double_conv_layer(up_128, 3, 4*64, dropout_rate, batch_norm)
# UpRes 8
gating_256 = gating_signal(up_conv_128, 2*64, batch_norm)
att_256 = attention_block(conv_256, gating_256, 2*64)
up_256 = layers.UpSampling2D(size=(2, 2), data_format="channels_last")(up_conv_128)
up_256 = layers.concatenate([up_256, att_256], axis=axis)
up_conv_256 = double_conv_layer(up_256, 3, 2*64, dropout_rate, batch_norm)
# UpRes 9
gating_512 = gating_signal(up_conv_128, 64, batch_norm)
att_512 = attention_block(conv_512, gating_512, 64)
up_512 = layers.UpSampling2D(size=(2, 2), data_format="channels_last")(up_conv_256)
up_512 = layers.concatenate([up_512, att_512], axis=axis)
up_conv_512 = double_conv_layer(up_512, 3, 64, dropout_rate, batch_norm)

# 1*1 convolutional layers
# valid padding
# batch normalization
# sigmoid nonlinear activation
conv_final = layers.Conv2D(NUM_CLASSES, kernel_size=(1,1))(up_conv_512)
conv_final = layers.BatchNormalization(axis=axis)(conv_final)
conv_final = layers.Activation('sigmoid')(conv_final)

# Model integration
model = models.Model(inputs, conv_final, name="AttentionResUNet")
return model

input_shape=(512,512,3)
model=Attention_ResUNet( input_shape, NUM_CLASSES=1,dropout_rate=0.0, batch_norm=True)
model.summary()
The code for training:
import os
import numpy as np
import tensorflow as tf
from tensorflow.keras import backend as K
os.environ[“TF_CPP_MIN_LOG_LEVEL”] = “2” #set to 1 for warnings and errors
import numpy as np
import cv2
import keras
import keras.utils
from glob import glob
from sklearn.utils import shuffle
import tensorflow as tf
from tensorflow.keras.callbacks import ModelCheckpoint, CSVLogger, ReduceLROnPlateau, EarlyStopping, TensorBoard
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.metrics import Recall, Precision
H = 512
W = 512
from focal_loss import BinaryFocalLoss #for tough to classify segement class
def create_dir(path):
“”" Create a directory. “”"
if not os.path.exists(path):
os.makedirs(path)
def shuffling(x, y):
x, y = shuffle(x, y, random_state=42)
return x, y
def load_data(path):
x = sorted(glob(os.path.join(path, “image”, “.png")))
y = sorted(glob(os.path.join(path, “mask”, ".png”)))
return x, y
def read_image(path):
path = path.decode()
x = cv2.imread(path, cv2.IMREAD_COLOR)
x = x/255.0
x = x.astype(np.float32)
return x
def read_mask(path):
path = path.decode()
x = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
x = x/255.0

x = x > 0.5
x = x.astype(np.float32)
x = np.expand_dims(x, axis=-1)
return x

def tf_parse(x, y):
def _parse(x, y):
x = read_image(x)
y = read_mask(y)
return x, y
x, y = tf.numpy_function(_parse, [x, y], [tf.float32, tf.float32])
x.set_shape([H, W, 3])
y.set_shape([H, W, 1])
return x, y

def tf_dataset(x, y, batch=8):
dataset = tf.data.Dataset.from_tensor_slices((x, y))
dataset = dataset.map(tf_parse)
dataset = dataset.batch(batch)
dataset = dataset.prefetch(10)
return dataset
if name == “main”:
“”" Seeding “”"
np.random.seed(42)
tf.random.set_seed(42)
""" Directory for storing files """
create_dir("files")

""" Hyperparameters """
batch_size = 2
lr = 0.002
num_epochs = 60
model_path = os.path.join("files", "model.h5")
csv_path = os.path.join("files", "data.csv")

""" Dataset """
train_path = os.path.join("/content/drive/MyDrive/Data_brain/train/")
valid_path = os.path.join("/content/drive/MyDrive/Data_brain/test/")

train_x, train_y = load_data(train_path)
train_x, train_y = shuffling(train_x, train_y)
valid_x, valid_y = load_data(valid_path)

print(f"Train: {len(train_x)} - {len(train_y)}")
print(f"Valid: {len(valid_x)} - {len(valid_y)}")

train_dataset = tf_dataset(train_x, train_y, batch=batch_size)
valid_dataset = tf_dataset(valid_x, valid_y, batch=batch_size)

""" Model """
model = Attention_ResUNet(input_shape)
metrics = [jacard_coef, Recall(), Precision()]
model.compile(loss=BinaryFocalLoss(gamma=2), optimizer=Adam(lr), metrics=metrics)

callbacks = [
    ModelCheckpoint(model_path, verbose=1, save_best_only=True),
    #ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=10, min_lr=1e-7, verbose=1),
    CSVLogger(csv_path),
    TensorBoard(),
    #EarlyStopping(monitor='val_loss', patience=50, restore_best_weights=False),
]

model.fit(
    train_dataset,
    epochs=num_epochs,
    validation_data=valid_dataset,
    callbacks=callbacks,
    shuffle=False)

I am getting the following error while training the model(the results are absurd):
Train: 1280 - 1280
Valid: 32 - 32
/usr/local/lib/python3.7/dist-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
category=CustomMaskWarning)
Epoch 1/60
640/640 [==============================] - 279s 414ms/step - loss: 0.0857 - jacard_coef: 0.0047 - recall: 0.0555 - precision: 0.0049 - val_loss: 0.0365 - val_jacard_coef: 0.0044 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00
Epoch 00001: val_loss improved from inf to 0.03647, saving model to files/model.h5
Epoch 2/60
640/640 [==============================] - 263s 411ms/step - loss: 0.0235 - jacard_coef: 0.0045 - recall: 0.0000e+00 - precision: 0.0000e+00 - val_loss: 0.0159 - val_jacard_coef: 0.0043 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00
Epoch 00002: val_loss improved from 0.03647 to 0.01592, saving model to files/model.h5
Epoch 3/60
39/640 [>…] - ETA: 4:05 - loss: 0.0159 - jacard_coef: 0.0045 - recall: 0.0000e+00 - precision: 0.0000e+00

Context:
Hi Aleena,
It’s a little bit hard to understand your question. Can you rephrase a little bit? maybe highlight the key parts?


A TF repository for point-cloud segmentation

Question:
I wanted to share a project I have been collaborating on with Soumik 1.
We take the problem of segmenting 3D point clouds that are important for modeling geometric properties from data. Today, we are delighted to open-source our repository 4 that implements the PointNet [1] model family for this purpose. We provide TensorFlow implementations with full support for TPUs and distributed training with mixed-precision (for GPUs). We provide models pre-trained on the four categories of the ShapeNet core dataset [2]. Here’s also a blog post 2 we have prepared to make it easier for getting started.

car964×525 94.9 KB

airplane966×525 74 KB

As always, don’t hesitate to reach out if you have any questions.
References:
[1] https://arxiv.org/abs/1612.00593 2
[2] https://shapenet.org/ 2

Context: This is great! I’m always interested in deep learning for new domains and data types.


Question:
Hello all,
I have a question about which solution will fits this case the best:
Case:
I have 3 kinds of text blocks: ingredients, preparation and dosage,  and want to classify this types
I have a lot data that are already categorized for training
I hope someone have some papers, GitHub links or even better experience for this case.
best regards !!

Context:
You can start with this tutorial: Classify structured data with feature columns  |  TensorFlow Core 2
It demonstrates how to deal with various data types.


A bit of attention for tfjs #3835 (angles from face mesh)

Question:
There is community-ticket tfjs #3835 1 - it asks to add rotation angle for face mesh.
I have submitted MR for it #844 1
I surely understand tfjs team has it’s own priorities, however my MR dangles there for already two weeks.
I would be glad if some response would be given, so I can continue with MR.

Context: Thank you for reminding us. This is a highly sought-after feature, we are working on it. Replied the ticket.


False prediction in-between frames w/ Fast RCNN object detection model

Question:
Hi,
I trained a Fast RCNN Model to detect water puddle, and the model predicted well.  However, there is an issue with the model on decoding a video stream running at 30fps.  As shown in the attached images,
frame# 476 - detected a puddle with 100% confidence level
frame# 477 - did not detect anything
frame# 478 - detected the same puddle again at 100% confidence level.
I would like to know if anyone has similar experience with Fast RCNN model, and what did you do to fix it?
FYI, I also did training with two other models, MobileNet v2 SSD and ResNet.  These two models gave gradual prediction results (conf. level fluctuates) as camera is panned over the subject.  Fast RCNN behaves erratically, for the most part, confidence level of the detected object is either > 98% or close to zero.  Please share if there is a way to fix this!
frame 476: frame476.jpg - Google Drive 1
frame 477: frame477.jpg - Google Drive 1

Context:
here is the third image:
frame 478: frame478.jpg - Google Drive 1


Question:
I’m looking into implementing high-quality image-processing operations using TF. For example, I’d like to have a higher-quality downsampling method, like Lanczos as a TF model. Please forward any references to this sort of work you are aware of.
For example, a basic Gaussian blur can be implemented by passing a custom-width kernel to tf.conv2d() (I’m using TFJS). This works great, but has the expected issues along the image boundary. Production-quality image processing tools solve this edge problem in one of a few ways, typically by adjusting the kernel weights outside the image to zero. However, I’m not experienced enough at how to set different kernels along the image boundaries.
Can anyone provide some tips?
For more context, here’s code that does a simple NxN Gaussian blur, without handling the borders. I’d love to figure out how to enhance this code to provide different kernels along the boundary rows and columns to do a better job of handling the edges (ie. not blending with zero).
const lanczos = (x, a) => {
  if (x === 0) return 1
  if (x >= -a && x < a) {
    return (a * Math.sin(Math.PI * x) * Math.sin(Math.PI * (x / a))) / (Math.PI * Math.PI * x * x)
  }
  return 0
}

const gaussian = (x, theta = 1 /* ~ -3 to 3 */) => {
  const C = 1 / Math.sqrt(2 * Math.PI * theta * theta)
  const k = -(x * x) / (2 * theta * theta)
  return C * Math.exp(k)
}

const filters = {
  Lanczos3: x => lanczos(x, 3),
  Lanczos2: x => lanczos(x, 2),
  Gaussian: x => gaussian(x, 1),
  Bilinear: () => 1,
  Nearest: () => 1,
}

const normalizedValues = (size, filter) => {
  let total = 0
  const values = []
  for (let y = -size; y <= size; ++y) {
    const i = y + size
    values[i] = []
    for (let x = -size; x <= size; ++x) {
      const j = x + size
      values[i][j] = []
      const f = filter(x) * filter(y)
      total += f
      for (let c = 0; c < 3; ++c) {
        values[i][j][c] = [ f, f, f ]
      }
    }
  }
  const kernel = values.map(row => row.map(col => col.map(a => a.map(b => b / total))))
  // for (let x = -size; x <= size; ++x) values[x + size] = filter(x)
  // const kernel = tf.einsum('i,j->ij', values, values)
  // const sum = tf.sum(values)
  const normalized = tf.div(kernel, total * 3)
  return normalized
}

const frame = async (tensor, args) => {
  const filter = filters[args.filter]
  // const [ height, width ] = tensor.shape
  // const res = args.resolution === 'Source' ? [ width, height ] : resolutions[args.resolution]
  // const strides = [ width / res[0], height / res[1] ]
  const { zoom, kernelWidth } = args
  const strides = Math.max(1, zoom)
  const size = Math.max(3, kernelWidth) * strides
  const kernel = normalizedValues(size, filter)
  const pad = 'valid' // sample to the edge, even when filter extends beyond image
  const dst = tf.conv2d(tensor, kernel, strides, pad)
  return { tensor: dst }
}

Context:
Dan_Wexler:

I’d love to figure out how to enhance this code to provide different kernels along the boundary rows and columns to do a better job of handling the edges (ie. not blending with zero)

Could you be more specific here?
You always have the option to slice the image into (overlapping): center, edges, corners. Apply the Conv to the center portion, apply modified convs to the edges and corners. And then 2d stack them back together. But there may be a short-cut depending on how you plan to modify the kernel at the edges.


Is it possible to crack human idea using tensorflow 2.5 or 2.6

Question: the benefits of this model will be in medical for example coma

Context: What is your goal? Can you make an example about your input and expected output?


Trying to use Tracker in Movenet Pose detector

Question:
Hi i`m trying to make a motion tracking application with  Movenet in React Native
Confirmed keypoints are detected and shown up on console but having trouble to enable tracker
How can I enable built in keypoints tracker in Movenet???
Attached my source code below
import React, { useState, useEffect, useCallback, useMemo } from ‘react’;
import { View, StyleSheet, Platform, TouchableOpacity, Text } from ‘react-native’;
import Icon from ‘react-native-vector-icons/Ionicons’
import { Colors } from ‘react-native-paper’;
import { Camera } from ‘expo-camera’;
import * as tf from ‘@tensorflow/tfjs’;
import {cameraWithTensors} from ‘@tensorflow/tfjs-react-native’;
import * as poseDetection from ‘@tensorflow-models/pose-detection’;
import ‘@tensorflow/tfjs-backend-webgl’;
import ‘@mediapipe/pose’;
let coords = []
export const CameraView = () => {
const [hasPermission, setHasPermission] = useState(null);

const [poseDetector, setPoseDetector] = useState(null);

const [frameworkReady, setFrameworkReady] = useState(false);

const backCamera = Camera.Constants.Type.back

const frontCamera = Camera.Constants.Type.front

const [camType, setCamType] = useState(backCamera)

const TensorCamera = cameraWithTensors(Camera);

let requestAnimationFrameId = 0;

const textureDims = Platform.OS === "ios"? { width: 1080, height: 1920 } : { width: 1600, height: 1200 };

const tensorDims = { width: 152, height: 200 };

const iconPressed = useCallback(() => camType === backCamera? setCamType(frontCamera):setCamType(backCamera),[camType])

const model = poseDetection.SupportedModels.MoveNet;

const detectorConfig = {

    modelType: poseDetection.movenet.modelType.MULTIPOSE_LIGHTNING,

    enableTracking: true,

    trackerType: poseDetection.TrackerType.Keypoint,

    trackerConfig: {maxTracks: 4,

        maxAge: 1000,

        minSimilarity: 1,

        keypointTrackerParams:{

            keypointConfidenceThreshold: 1,

            keypointFalloff: [],

            minNumberOfKeypoints: 4

        }

    }

}

const detectPose = async (tensor) =>{

    if(!tensor) return

    const poses = await poseDetector.estimatePoses(tensor)

    if (poses[0] !== undefined) {

        const points = poses[0].keypoints.map(point => [point.x,point.y,point.name])

        console.log(points)

        coords = points

    } else {

        coords = []

    }

    ///console.log(coords)

}

const handleCameraStream = (imageAsTensors) => {

    const loop = async () => {

        const nextImageTensor = await imageAsTensors.next().value;

        await detectPose(nextImageTensor);

        requestAnimationFrameId = requestAnimationFrame(loop);

    };

    if (true) loop();

  }

useEffect(() => {

if(!frameworkReady) {

    ;(async () => {

    const { status } = await Camera.requestPermissionsAsync();

    console.log(`permissions status: ${status}`);

    setHasPermission(status === 'granted');

    await tf.ready();

    setPoseDetector(await poseDetection.createDetector(model, detectorConfig))

    setFrameworkReady(true);

    })();

}

}, []);

useEffect(() => {

    return () => {

    cancelAnimationFrame(requestAnimationFrameId);

    };

}, [requestAnimationFrameId]);

return(

    <View style={styles.cameraView}>

        <TensorCamera

            style={styles.camera}

            type={camType}

            zoom={0}

            cameraTextureHeight={textureDims.height}

            cameraTextureWidth={textureDims.width}

            resizeHeight={tensorDims.height}

            resizeWidth={tensorDims.width}

            resizeDepth={3}

            onReady={(imageAsTensors) => handleCameraStream(imageAsTensors)}

            autorender={true}

        >

        </TensorCamera>

        <TouchableOpacity style={[styles.absoluteView]} activeOpacity={0.1}>

            <Icon name="camera-reverse-outline" size={40} color="white" onPress={iconPressed}/>

        </TouchableOpacity>

    </View>

)

}
const styles = StyleSheet.create({
camera:{flex:1},

cameraView:{flex:1},

absoluteView:{

    position:'absolute',

    right:30,

    bottom: Platform.select({ios:40, android:30}),

    padding: 10,

},

tracker:{

    position:'absolute',

    width:10,

    height:10,

    borderRadius:5,

    backgroundColor: Colors.blue500

}

})

Context:
Hi @11130 , you need to lower minSimilarity, the semantics of this field is that the similarity between current pose and tracked pose, if their similarity is larger than minSimlarity, then we consider them as the same preson. 1 is the largest possible score for similarity, if you set it to 1, then that means we only consider this is the same person if current pose is exactly same as before. The default minSimilarity is 0.15. If you want to use the default, you can omit this field. Same for keypointConfidenceThreshold, the default is 0.3. Also you need to set values for keypointFalloff, default is [
0.026, 0.025, 0.025, 0.035, 0.035, 0.079, 0.079, 0.072, 0.072, 0.062,
0.062, 0.107, 0.107, 0.087, 0.087, 0.089, 0.089
]. I suggest just use default, to use default, just omit keypointTrackerParams.


Implementing Sparse Ternary Compression inside Tensorflow Federated simple_fedavg example

Question:
Hi all,
I’m actually trying to implement an Algorithm that i found by reading the paper Robust and Communication-Efficient Federated Learning from Non-IID Data by [Simon Wiedemann] [Klaus-Robert Müller] [Wojciech].
I wanted to try to implement it inside the simple_fedavg offered by Tensorflow Federated. I have actually already created the algorithm and seems to works fine in test case, the real problem is to put it inside the simple_fedavg project. I don’t get where i could change what the client send to the server and what the server expect to recieve.
So, basically, from client_update i don’t want to send weights_delta, but instead i want to send a simple list like [ [list of negatives indexes] [list of positives indexes] [average value] ], then on the server side i will recreate the weights like explained in the paper. But i can’t understand how to change this behaviour.
English is not my main language, so i hope i have explained the problem good enough.
  test = weights_delta.copy()
  for index in range(len(weights_delta)):
      original_shape = tf.shape(weights_delta[index])
      tensor = tf.reshape(test[index], [-1])
      negatives, positives, average = test_stc.stc_compression(tensor, sparsification_rate)
      test[index] = test_stc.stc_decompression(negatives, positives, average, tensor.get_shape().as_list(), original_shape)
      test[index] = test_stc.stc_compression(tensor, sparsification_rate)

  client_weight = tf.cast(num_examples, tf.float32)
  return ClientOutput(test, client_weight, loss_sum / client_weight)

This is the behaviour that i would like, stc_compression return a tuple. Then i would like to access to each “test” variable sent from a client inside the server and recreate all the weights.

Context:
@lgusm Can we have some TF federated team member subscribed to this federated tag?
Thanks


Efficientdet vs ResNet: Resnet outperforms Efficientdet-B3

Question:
So I tried to swap out ResNet for Efficientdet-B3 in the Eager Few Shot OD Training TF2 tutorial 3.
Now, based on all the positive feedback Efficientdet got I am very surprised that ResNet outperformed Efficientdet on this tutorial. In total Efficientdet got trained on 1700 batches in the tutorial, while I ran ResNet through the standard batch size of 100.
Efficientdet-B3 for the last 1000 batches I run of a total of 1700:
batch 950 of 1000, loss=0.21693243
batch 955 of 1000, loss=0.18070191
batch 960 of 1000, loss=0.1715184
batch 965 of 1000, loss=0.23656633
batch 970 of 1000, loss=0.16813375
batch 975 of 1000, loss=0.23602965
batch 980 of 1000, loss=0.14852181
batch 985 of 1000, loss=0.18400437
batch 990 of 1000, loss=0.22741726
batch 995 of 1000, loss=0.20477971
Done fine-tuning!

ResNet for 100 batches:
batch 0 of 100, loss=1.1079819
batch 10 of 100, loss=0.07644452
batch 20 of 100, loss=0.08746071
batch 30 of 100, loss=0.019333005
batch 40 of 100, loss=0.0071129226
batch 50 of 100, loss=0.00465827
batch 60 of 100, loss=0.0041421074
batch 70 of 100, loss=0.0026128457
batch 80 of 100, loss=0.0023376464
batch 90 of 100, loss=0.002139934
Done fine-tuning!

Why does Efficientdet need so much more training time than ResNet, is it due to that the number of parameters is only about 12 mill for Efficientdet-B3 (the one I tested) and about 25 mill for the ResNet50? Or are their other reasons?
The end result (the .gif at the end of the tutorial) also shows a huge different in accuracy, where ResNet performs much better.
Thanks for any input!

Context: hello, when you mean “outperform” are you referring to the evaluation/test metrics? you should always monitor the gap between the training and validation losses, they should behave similar and then evaluate to see the final performance on a test set. Otherwise ResNet might be just overfitting the data and that’s why you get a extremely small loss, are these losses you posted training losses?


[Research ] Nowcasting the Next Hour of Rain (by DeepMind)

Question:
Deepmind

Nowcasting 3

  Our latest research and state-of-the-art model advances the science of Precipitation Nowcasting.

GitHub: https://github.com/deepmind/deepmind-research/tree/master/nowcasting 14
Colab: Google Colab 14
Paper: Skilful precipitation nowcasting using deep generative models of radar | Nature 9

Recently introduced deep learning methods use radar to directly predict future rain rates, free of physical constraints5,6. While they accurately predict low-intensity rainfall, their operational utility is limited because their lack of constraints produces blurry nowcasts at longer lead times, yielding poor performance on rarer medium-to-heavy rain events. Here we present a deep generative model for the probabilistic nowcasting of precipitation from radar that addresses these challenges.

https://lh3.googleusercontent.com/8PmJxGsCJ01Usa4ZN5cRKng8bIJMVAYHQwmzwBe5mZqWMazGljujwUplM0VCP1ZEzghp6Ie65gJALkLWzR2fGLopN8bIAKbFBvc4zJi4HzNHR4OX3Vc=w2048-rw-v1(image larger than 4096KB) 6

Context: this is super cool and there’s a colab for you to try the model: https://colab.sandbox.google.com/github/deepmind/deepmind-research/blob/master/nowcasting/Open_sourced_dataset_and_model_snapshot_for_precipitation_nowcasting.ipynb#scrollTo=wFD0zFFyuHzH 9


Question: I am stucking with one problem while defining one model in tf2. I am using one dense layer with a softmax activation function. Now I want to extract the index with the highest probability value for that softmax layer output . So that I can use that index for later layer definition while building the model. Please help me out to implement the above problem, Waiting for your quick reply.

Context:
tf.math.argmax is the tensorflow operations which will allow you to do this. If you have a Keras model you can use a Custom layers, as described in the tutorial: Custom layers  |  TensorFlow Core 1
The final code will look something like:
class IndexOfMaxLayer(tf.keras.layers.Layer):
  def __init__(self):
    super(IndexOfMaxLayer, self).__init__()

  def call(self, inputs):
    return tf.math.argmax(inputs)

Now you can use this code after whichever dense / softmax layer you want to extract the max value from.


Question:
As of today, different variants of MLP-Mixers [1] are now available on TensorFlow Hub. Below are the details:

Models: https://tfhub.dev/sayakpaul/collections/mlp-mixer/1 10

Code: https://git.io/JzR68 7

Here is some example usage:

References:
[1] MLP-Mixer: An all-MLP Architecture for Vision by Tolstikhin et al. 4

Context: Well done Sayak!!! Great work!!


Tensorflow problem: The loss return None, and show Error message:Attempting to capture an EagerTensor without building a function

Question:
Hi guys, I try to implement the model for tensorflow2.5.0, but when I run the model, its print my loss return ‘none’, and show the error message: “RuntimeError: Attempting to capture an EagerTensor without building a function”.

image938×543 28.2 KB

Hope guys help me find the bug.
This is my model code:
encode model:

image936×1257 153 KB

decode model:

image825×806 90.4 KB

discriminator model:

image1171×687 79.6 KB

training step:

image1512×1142 219 KB

loss function:

image884×65 7.83 KB

image1288×130 17.7 KB

image737×178 22.7 KB

image890×58 10.1 KB

image946×56 9.48 KB

There is I have check:

I checked my dataset. There is not none data.
I checked my loss function, there is no nd.array, I change in tf.tensor.

This is my first time ask question on the website, if I need provide other code information to solve problem, I will upload. Thanks.

Context: If you can share a running Colab to reproduce this it could be ideal.


How can I predict the next month with time series?

Question:
Hello everyone, I have a question related to time series. I have a year of data of a specific machine on its fuel consumption, I would like to predict an approximate total consumption for the next month (the  value). Following the example of Tensorflow, the prediction works for me, but only for periods already known (checking the prediction). but I have modified the Window of time and I have not been able to predict a future date.
I am following this procedure
https://www.tensorflow.org/tutorials/structured_data/time_series 17
any recommendation or any idea? I really precited!
Thanks , Ricardo.

Context:
Hey @Ricardo can you share the requirement that specifics about your needs, especially with the data?
Modeling time series issues are based on how you outline the samples and labels. There are basically two different ways roughly from high level views:

Single step to predict
Multi steps to predict.

This is my point of view.
Actually it is about how you want to choose the time window to train your model.


Question:
Ever wanted to use Vision Transformers with TFHub and Keras? Well, pull your socks up now and get started. 16 different models are available for classification and fine-tuning. More details:

      github.com

GitHub - sayakpaul/ViT-jax2tf: This repository hosts code for converting the... 68

  This repository hosts code for converting the original Vision Transformer models (JAX) to TensorFlow.  - GitHub - sayakpaul/ViT-jax2tf: This repository hosts code for converting the original Vision...

The story does not end here. Using this notebook you can convert any supported model from the AugReg pool (> 50,000 models!) and use that inside TFHub and Keras: https://colab.research.google.com/github/sayakpaul/ViT-jax2tf/blob/main/conversion.ipynb 20.

carbon1408×616 75.1 KB

Context:
Well done Sayak! great work as usual!!!
Just to complement, link to the the TFHub collection: TensorFlow Hub 21


How to calculate FLOPs of transformer in tensorflow?

Question:
I know that
    flops = tf.profiler.profile(graph, options=tf.profiler.ProfileOptionBuilder.float_operation())

can calculate the FLOPs.
But where can I find the graph of transformer?
Please help me.

Context:
There Is a quite long thread for this in TF 2.x:

      github.com/tensorflow/tensorflow

      TF 2.0 Feature: Flops calculation 26

        opened

        Sep 25, 2019

          pzobel

        stat:awaiting tensorflower
        type:feature
        comp:tfdbg
        TF 2.0

    Please make sure that this is a feature request. As per our [GitHub Policy](…https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md), we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

**System information**
- TensorFlow version (you are using): TF 2.0 RC2
- Are you willing to contribute it (Yes/No):

**Describe the feature and the current behavior/state.**

I am missing the opportunity to compute the number of floating point operations of a tf.keras Model in TF 2.0.
In TF 1.x tf.profiler was available [see here](https://stackoverflow.com/questions/45085938) but I can find anything equivalent for TF 2.0 yet.

**Will this change the current api? How?**

**Who will benefit with this feature?**

Everbody interested in the computational complexity of a TensorFlow model.

**Any Other info.**


Model for sentiment analysis in Nigerian (and other) languages

Question:
Hi. Very new to this forum and more to Tensorflow, and I had a quick question that I hope someone can help me in.
Basically, I’m working with a set of languages (hausa, yoruba and igbo) that do not have a reliable sentiment analysis model to process text with - unless I missed something. What I want is to create a custom model for each of these languages where the model scores and returns the sentiment of a sentence as accurately as possible.
I’m not sure how to approach this. What I did first is got a training dataset where the text and its sentiment (human scored), vectorized the text and created a model (using an Embedding layer). The accuracy wasn’t the best but I don’t know if that is the way to continue. Selecting the right hyperparameters seems like a separate job on its own.
Can anyone recommend on how you might approach this? And if there’s documentation on how a sentiment analysis model using these languages (or any non-English) language is created?
Any help would be appreciated. Thanks.

Context:
Hi, welcome to the forum, Is this for
https://lacunafund.org/language-2020-awards/ 2


KerasTuner & TensorFlow Decision Forests

Question:
Hi,
Is there a way to use KerasTuner on tensorflow_decision_forests?
Any tutorial?
Thanks
Fadi

Context:
Out of interest I checked that the basic KerasTuner logic described in the tutorial here (Getting started with KerasTuner 5) works with decision forest model the same way as with neural networks.
def build_model(hp):
    """Function initializes the model and defines search space.
    :param hp: Hyperparameters
    :return: Compiled TensorFlow model
    """
    model = tfdf.keras.GradientBoostedTreesModel(
        num_trees=hp.Int('num_trees', min_value=10, max_value=510, step=50),
        max_depth=hp.Int('max_depth', min_value=3, max_value=16, step=1))
    model.compile(metrics=['accuracy'])
    return model

tuner = kt.RandomSearch(
    build_model,
    objective='val_loss',
    max_trials=5)

tuner.search(X_train, y_train, epochs=1, validation_data=(X_valid, y_valid))