Error when running “ninja all” to build glow

Question:
My Linux version is Ubuntu 18.04.
I followed a solution in this post 2 to avoid the “bundle error”. (In fact, I also encountered that error if I didn’t delete the line “bundle_with_multiple_entries”)
Then I ran the following command to generate the build config:
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DGLOW_WITH_BUNDLES=ON -DLLVM_DIR=/usr/lib/llvm-7/cmake/ -DCMAKE_C_COMPILER=/usr/lib/llvm-7/bin/clang -DCMAKE_CXX_COMPILER=/usr/lib/llvm-7/bin/clang++ -DLLVM_CONFIG=/usr/lib/llvm-7/bin/llvm-config ../glow

After that, I ran “ninja all” to build, but I encountered the following error:
2021-05-24 21:55:25 (9.48 MB/s) - ‘/home/user/build_Release/examples/bundles/resnet50/resnet50/init_net.pb’ saved [128070759/128070759]

[670/724] Generating profile.yml
FAILED: examples/bundles/lenet_mnist/profile.yml
cd /home/user/build_Release/examples/bundles/lenet_mnist && /home/user/build_Release/bin/image-classifier /home/user/glow/tests/images/mnist/*.png -i=0to1 -dump-profile=/home/user/build_Release/examples/bundles/lenet_mnist/profile.yml -m=/home/user/build_Release/examples/bundles/lenet_mnist/lenet_mnist -model-input-name=data
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0524 21:57:25.627862 24050 Partitioner.cpp:693] Profiling a model to be partitioned cross different backends. Each sub-network will be optimized and run on cpu backend.
LLVMIRGen: libjit.bc: error: Unknown attribute kind (60) (Producer: 'LLVM9.0.1' Reader: 'LLVM 7.1.0')
F0524 21:57:25.651914 24054 LLVMIRGen.cpp:216] Check failed: llmodule_.get() Unable to load the JIT library.
*** Check failure stack trace: ***
#0 0x0000000000ac799e llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/user/build_Release/bin/image-classifier+0xac799e)
#1 0x0000000000ac5ea1 llvm::sys::RunSignalHandlers() (/home/user/build_Release/bin/image-classifier+0xac5ea1)
#2 0x0000000000ac7cb1 SignalHandler(int) (/home/user/build_Release/bin/image-classifier+0xac7cb1)
#3 0x00007f742018297f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1297f)
#4 0x00007f741f47efb7 raise /build/glibc-S9d2JN/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:51:0
#5 0x00007f741f480920 abort /build/glibc-S9d2JN/glibc-2.27/stdlib/abort.c:79:0
#6 0x00007f7423a891f8 (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xa1f8)
#7 0x00007f7423a8b0cc google::LogMessage::Fail() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xc0cc)
#8 0x00007f7423a8cf32 google::LogMessage::SendToLog() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xdf32)
#9 0x00007f7423a8ac27 google::LogMessage::Flush() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xbc27)
#10 0x00007f7423a8d998 google::LogMessageFatal::~LogMessageFatal() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xe998)
#11 0x0000000000c17e2e glow::LLVMIRGen::initCodeGen() (/home/user/build_Release/bin/image-classifier+0xc17e2e)
#12 0x0000000000c3b2af glow::LLVMBackend::compileIRWithoutConstants(glow::IRFunction*) const (/home/user/build_Release/bin/image-classifier+0xc3b2af)
#13 0x0000000000c3bb0a glow::LLVMBackend::compile(glow::Function*, glow::BackendOptions const&) const (/home/user/build_Release/bin/image-classifier+0xc3bb0a)
#14 0x00000000025d6574 std::thread::_State_impl<std::thread::_Invoker<std::tuple<glow::Backend::compileFunctions(std::vector<glow::Function*, std::allocator<glow::Function*> >&, llvm::StringMap<glow::BackendOptions, llvm::MallocAllocator>&) const::$_0, (anonymous namespace)::PerCompilationThreadState*> > >::_M_run() (/home/user/build_Release/bin/image-classifier+0x25d6574)
#15 0x00007f741fb066de (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd6de)
#16 0x00007f74201776da start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#17 0x00007f741f56171e clone /build/glibc-S9d2JN/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95:0
Aborted (core dumped)
[687/724] Generating profile.yml
FAILED: examples/bundles/resnet50/profile.yml
cd /home/user/build_Release/examples/bundles/resnet50 && /home/user/build_Release/bin/image-classifier /home/user/glow/tests/images/imagenet/*.png -i=0to1 -dump-profile=/home/user/build_Release/examples/bundles/resnet50/profile.yml -m=/home/user/build_Release/examples/bundles/resnet50/resnet50 -model-input-name=gpu_0/data
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0524 21:57:26.608170 24047 Partitioner.cpp:693] Profiling a model to be partitioned cross different backends. Each sub-network will be optimized and run on cpu backend.
LLVMIRGen: libjit.bc: error: Unknown attribute kind (60) (Producer: 'LLVM9.0.1' Reader: 'LLVM 7.1.0')
F0524 21:57:26.623073 24058 LLVMIRGen.cpp:216] Check failed: llmodule_.get() Unable to load the JIT library.
*** Check failure stack trace: ***
#0 0x0000000000ac799e llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/user/build_Release/bin/image-classifier+0xac799e)
#1 0x0000000000ac5ea1 llvm::sys::RunSignalHandlers() (/home/user/build_Release/bin/image-classifier+0xac5ea1)
#2 0x0000000000ac7cb1 SignalHandler(int) (/home/user/build_Release/bin/image-classifier+0xac7cb1)
#3 0x00007f012a57597f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1297f)
#4 0x00007f0129871fb7 raise /build/glibc-S9d2JN/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:51:0
#5 0x00007f0129873920 abort /build/glibc-S9d2JN/glibc-2.27/stdlib/abort.c:79:0
#6 0x00007f012de7c1f8 (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xa1f8)
#7 0x00007f012de7e0cc google::LogMessage::Fail() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xc0cc)
#8 0x00007f012de7ff32 google::LogMessage::SendToLog() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xdf32)
#9 0x00007f012de7dc27 google::LogMessage::Flush() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xbc27)
#10 0x00007f012de80998 google::LogMessageFatal::~LogMessageFatal() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xe998)
#11 0x0000000000c17e2e glow::LLVMIRGen::initCodeGen() (/home/user/build_Release/bin/image-classifier+0xc17e2e)
#12 0x0000000000c3b2af glow::LLVMBackend::compileIRWithoutConstants(glow::IRFunction*) const (/home/user/build_Release/bin/image-classifier+0xc3b2af)
#13 0x0000000000c3bb0a glow::LLVMBackend::compile(glow::Function*, glow::BackendOptions const&) const (/home/user/build_Release/bin/image-classifier+0xc3bb0a)
#14 0x00000000025d6574 std::thread::_State_impl<std::thread::_Invoker<std::tuple<glow::Backend::compileFunctions(std::vector<glow::Function*, std::allocator<glow::Function*> >&, llvm::StringMap<glow::BackendOptions, llvm::MallocAllocator>&) const::$_0, (anonymous namespace)::PerCompilationThreadState*> > >::_M_run() (/home/user/build_Release/bin/image-classifier+0x25d6574)
#15 0x00007f0129ef96de (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd6de)
#16 0x00007f012a56a6da start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#17 0x00007f012995471e clone /build/glibc-S9d2JN/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95:0
Aborted (core dumped)
[703/724] Building CXX object tests/unittests/CMakeFiles/CPUOperatorTest.dir/OperatorTest.cpp.o2021-05-24 21:55:25 (9.48 MB/s) - ‘/home/user/build_Release/examples/bundles/resnet50/resnet50/init_net.pb’ saved [128070759/128070759]

[670/724] Generating profile.yml
FAILED: examples/bundles/lenet_mnist/profile.yml
cd /home/user/build_Release/examples/bundles/lenet_mnist && /home/user/build_Release/bin/image-classifier /home/user/glow/tests/images/mnist/*.png -i=0to1 -dump-profile=/home/user/build_Release/examples/bundles/lenet_mnist/profile.yml -m=/home/user/build_Release/examples/bundles/lenet_mnist/lenet_mnist -model-input-name=data
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0524 21:57:25.627862 24050 Partitioner.cpp:693] Profiling a model to be partitioned cross different backends. Each sub-network will be optimized and run on cpu backend.
LLVMIRGen: libjit.bc: error: Unknown attribute kind (60) (Producer: 'LLVM9.0.1' Reader: 'LLVM 7.1.0')
F0524 21:57:25.651914 24054 LLVMIRGen.cpp:216] Check failed: llmodule_.get() Unable to load the JIT library.
*** Check failure stack trace: ***
#0 0x0000000000ac799e llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/user/build_Release/bin/image-classifier+0xac799e)
#1 0x0000000000ac5ea1 llvm::sys::RunSignalHandlers() (/home/user/build_Release/bin/image-classifier+0xac5ea1)
#2 0x0000000000ac7cb1 SignalHandler(int) (/home/user/build_Release/bin/image-classifier+0xac7cb1)
#3 0x00007f742018297f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1297f)
#4 0x00007f741f47efb7 raise /build/glibc-S9d2JN/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:51:0
#5 0x00007f741f480920 abort /build/glibc-S9d2JN/glibc-2.27/stdlib/abort.c:79:0
#6 0x00007f7423a891f8 (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xa1f8)
#7 0x00007f7423a8b0cc google::LogMessage::Fail() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xc0cc)
#8 0x00007f7423a8cf32 google::LogMessage::SendToLog() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xdf32)
#9 0x00007f7423a8ac27 google::LogMessage::Flush() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xbc27)
#10 0x00007f7423a8d998 google::LogMessageFatal::~LogMessageFatal() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xe998)
#11 0x0000000000c17e2e glow::LLVMIRGen::initCodeGen() (/home/user/build_Release/bin/image-classifier+0xc17e2e)
#12 0x0000000000c3b2af glow::LLVMBackend::compileIRWithoutConstants(glow::IRFunction*) const (/home/user/build_Release/bin/image-classifier+0xc3b2af)
#13 0x0000000000c3bb0a glow::LLVMBackend::compile(glow::Function*, glow::BackendOptions const&) const (/home/user/build_Release/bin/image-classifier+0xc3bb0a)
#14 0x00000000025d6574 std::thread::_State_impl<std::thread::_Invoker<std::tuple<glow::Backend::compileFunctions(std::vector<glow::Function*, std::allocator<glow::Function*> >&, llvm::StringMap<glow::BackendOptions, llvm::MallocAllocator>&) const::$_0, (anonymous namespace)::PerCompilationThreadState*> > >::_M_run() (/home/user/build_Release/bin/image-classifier+0x25d6574)
#15 0x00007f741fb066de (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd6de)
#16 0x00007f74201776da start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#17 0x00007f741f56171e clone /build/glibc-S9d2JN/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95:0
Aborted (core dumped)
[687/724] Generating profile.yml
FAILED: examples/bundles/resnet50/profile.yml
cd /home/user/build_Release/examples/bundles/resnet50 && /home/user/build_Release/bin/image-classifier /home/user/glow/tests/images/imagenet/*.png -i=0to1 -dump-profile=/home/user/build_Release/examples/bundles/resnet50/profile.yml -m=/home/user/build_Release/examples/bundles/resnet50/resnet50 -model-input-name=gpu_0/data
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0524 21:57:26.608170 24047 Partitioner.cpp:693] Profiling a model to be partitioned cross different backends. Each sub-network will be optimized and run on cpu backend.
LLVMIRGen: libjit.bc: error: Unknown attribute kind (60) (Producer: 'LLVM9.0.1' Reader: 'LLVM 7.1.0')
F0524 21:57:26.623073 24058 LLVMIRGen.cpp:216] Check failed: llmodule_.get() Unable to load the JIT library.
*** Check failure stack trace: ***
#0 0x0000000000ac799e llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/user/build_Release/bin/image-classifier+0xac799e)
#1 0x0000000000ac5ea1 llvm::sys::RunSignalHandlers() (/home/user/build_Release/bin/image-classifier+0xac5ea1)
#2 0x0000000000ac7cb1 SignalHandler(int) (/home/user/build_Release/bin/image-classifier+0xac7cb1)
#3 0x00007f012a57597f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1297f)
#4 0x00007f0129871fb7 raise /build/glibc-S9d2JN/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:51:0
#5 0x00007f0129873920 abort /build/glibc-S9d2JN/glibc-2.27/stdlib/abort.c:79:0
#6 0x00007f012de7c1f8 (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xa1f8)
#7 0x00007f012de7e0cc google::LogMessage::Fail() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xc0cc)
#8 0x00007f012de7ff32 google::LogMessage::SendToLog() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xdf32)
#9 0x00007f012de7dc27 google::LogMessage::Flush() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xbc27)
#10 0x00007f012de80998 google::LogMessageFatal::~LogMessageFatal() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xe998)
#11 0x0000000000c17e2e glow::LLVMIRGen::initCodeGen() (/home/user/build_Release/bin/image-classifier+0xc17e2e)
#12 0x0000000000c3b2af glow::LLVMBackend::compileIRWithoutConstants(glow::IRFunction*) const (/home/user/build_Release/bin/image-classifier+0xc3b2af)
#13 0x0000000000c3bb0a glow::LLVMBackend::compile(glow::Function*, glow::BackendOptions const&) const (/home/user/build_Release/bin/image-classifier+0xc3bb0a)
#14 0x00000000025d6574 std::thread::_State_impl<std::thread::_Invoker<std::tuple<glow::Backend::compileFunctions(std::vector<glow::Function*, std::allocator<glow::Function*> >&, llvm::StringMap<glow::BackendOptions, llvm::MallocAllocator>&) const::$_0, (anonymous namespace)::PerCompilationThreadState*> > >::_M_run() (/home/user/build_Release/bin/image-classifier+0x25d6574)
#15 0x00007f0129ef96de (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd6de)
#16 0x00007f012a56a6da start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#17 0x00007f012995471e clone /build/glibc-S9d2JN/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95:0
Aborted (core dumped)
[703/724] Building CXX object tests/unittests/CMakeFiles/CPUOperatorTest.dir/OperatorTest.cpp.o

Could anyone help me identify what’s the problem here? Thanks in advance!

Context:
LLVMIRGen: libjit.bc: error: Unknown attribute kind (60) (Producer: ‘LLVM9.0.1’ Reader: ‘LLVM 7.1.0’)

It seems like there is a mismatch between llvm versions on your machine – can you try to ensure you’re using the same llvm everywhere?


Error message: Failed to parse the network descriptor

Question:
When I execute this command " ./image-classifier tests/images/imagenet/cat_285.png -use-imagenet-normalization -image-mode=0to1 -m=resnet50 -model-input-name=gpu_0/data -dump-llvm-ir" , this error message "

image839×246 8.25 KB

" occur.
why this error occur??

Context: I think this is because you don’t have the directory resnet50/ containing the model protobuf files in the directory you’re running the command from.


Question:
The PyTorch Hackathon is back!

Taking place virtually from Sept 8 - Nov 3.
Enter to compete and connect with your worldwide PyTorch community across 3 project categories:

PyTorch Dev Tools
Web/Mobile Applications
PyTorch Responsible AI Dev Tools

Register here: https://pytorch2021.devpost.com 21

Context:
Hello @suraj.pt,
My name is Emil Bogomolov. I am a research engineer at Skolkovo Institute of Science and Technology (https://scholar.google.com/citations?user=g7jgY8MAAAAJ 2) .
Apart from the research I am active contributor to PyTorch (zetyquickly (Emil Bogomolov) · GitHub 3) and to PyTorch community (Profile - zetyquickly - PyTorch Forums). I really love this framework and urge to improve it and help others to use.
Also I do mentoring and judging at hackathons such as MLH (https://mlh.io/ 2) and Cifrovoy Proryv (https://leadersofdigital.ru/)
I’d like to offer myself as a judge to your upcoming hackathon it would be a big pleasure.
Thank you!


Question:
I am doing a face recognition app using facenet pytorch (https://github.com/timesler/facenet-pytorch 28) in python using two methods.
1st method  Code -
resnet = InceptionResnetV1(pretrained='vggface2').eval()
mtcnn = MTCNN(image_size=96)

img = Image.open(image_path)
image_prep = mtcnn(img)
plt.figure()
plt.axis('off')
plt.imshow(image_prep.permute(1, 2, 0))
if image_prep is not None:
  image_embedding = resnet(image_prep.unsqueeze(0))

In this code, I extract the face from the given images, and get the 512 encodings for recognizing faces.
In this example, I used two different faces, and plot the distance between the faces
        a.jpg       b.jpg
a.jpg   0.000000    1.142466
b.jpg   1.142466    0.000000

It works well…
2nd method  Code-
 img = Image.open(image)
 boxes, probs = mtcnn.detect(img) # Gives the coordinates of the face in the given image
 face = img.crop((boxes[0][0], boxes[0][1], boxes[0][2], boxes[0][3])) # Cropping the face
 plt.figure()
 plt.imshow(face)
 pil_to_tensor = transforms.ToTensor()(face).unsqueeze_(0) # Converting to tensor type
 image_embedding = resnet(pil_to_tensor)

In this code, I used to get the coordinates of the face, first and then the embeddings. The distance between the two faces -
        a.jpg       b.jpg
a.jpg   0.000000    0.631094
b.jpg   0.631094    0.000000

In 1st method, I directly feed the image to the mtcnn and gets better result, the distance between the two faces are more than 1.0. In, 2nd method, I gets the coordinates of the faces using  mtcnn.detect() , cropped the face from the given image, and feed to resnet. This method gives less distance between the two different faces.
Then, I find the reason for why the 1st method performs well than the 2nd method by plotting the result(face) before feeding to resnet.
In 2nd method, I feed the faces same as given in the input image(clear image) by cropping the faces using  mtcnn.detect() .
But, In 1st method, I directly gives the input to  mtcnn(img)  which returns the tensor of the faces in dark than in input image. This darker image is not a clear image(area around eyes are darker, I tested with many photos), Can’t able to see the eyes clearly. This is the reason, the 1st method shows higher distance between two faces.
My doubt is, Why mtcnn returns the tensor(faces) in dark, How to solve it, Help me in this issue,
Thanks

Context: Could you show an example of the “dark” output and what you would expect it to look like?


Anyone working with a data pipeline of CPU -> GPU? I am developing a library of methods for faster transfer to GPU. In some cases, 370x faster than used Pytorch’s Pinned CPU Tensors

Question:
Let me know what your pipeline is and I’ll try to add methods for it. Just show me your code.
I am developing methods for fast transfer from CPU and GPU, and currently coding the methods for it. Show me your code (A Colab notebook would be really helpful) and I’ll see how to incorporate the library into it, for faster data transfer.

Context:
@Santosh-Gupta Hi, thanks for your work and proposal. Actually I am very interested on tools to speed up CPU-GPU-CPU data transfers. I have written the details of my application in this post 18. There you will find details on what I tried and access to the code.
Basically, im trying to do real-time control @100hz on constrained devices (Jetson Nano) using pytorch for computation of the control laws on GPU. I am sure there are several bottlenecks but one of them is the CPU-GPU and GPU-CPU data transfers.
I am starting to rewrite the code to use torch JIT, but I am not quite sure about the speed up improvements. I will appreciate your support on trying out SpeedTorch (of which I read about on your post 3) and hopefully get some nice performance increase.


Introducing SpeedTorch: 4x speed CPU->GPU transfer, 110x GPU->CPU transfer

Question:
34

      GitHub

Santosh-Gupta/SpeedTorch 257

Library for fastest pinned CPU <-> GPU transfer . Contribute to Santosh-Gupta/SpeedTorch development by creating an account on GitHub.

This is my submission to the Pytorch Hackathon. It’s a library I made for Pytorch, for fast transfer between pinned CPU tensors and GPU pytorch variables. The inspiration came from needed to train large number of embeddings, which don’t all fit on GPU ram at a desired embedding size, so I needed a faster CPU <-> GPU transfer method. This also allows using any optimizer for sparse training, since every embedding contained in the Pytorch embedding variable receives an update, previously only Pytorch’s SGD, Adagrad, and SparseAdam were suitable for such training.
In addition to augmenting parameter sizes, you can use to increase the speed of which data on your CPU is transferred to Pytorch Cuda variables.
Also, SpeedTorch’s GPU tensors are also overall faster then Pytorch cuda tensors, when taking into account both transferring two and from (overall 2.6x faster). For just transfering to a Pytorch Cuda, Pytorch is still faster, but significantly slower when transfering from a Pytorch Cuda variable.
I have personally used this to nearly double the embedding size of embeddings in two other projects, by holding half the parameters on CPU. The training speed is decent thanks to the fast CPU<->GPU exchange.

      GitHub

Santosh-Gupta/Research2Vec2 7

Updated from the original Research2vec project. Contribute to Santosh-Gupta/Research2Vec2 development by creating an account on GitHub.

      GitHub

Santosh-Gupta/Lit2Vec2 4

Contribute to Santosh-Gupta/Lit2Vec2 development by creating an account on GitHub.

There’s a bit of a learning curve for the very first time getting started with it, so as soon as you run into any sort of friction, feel free to ask a question on the project gitter
https://gitter.im/SpeedTorch 92
And I’ll answer them.

           2

Context:
Interesting work. This can speed-up operations like:
cpu_tensor[idx] = gpu_tensor.cpu()

and
gpu_tensor = cpu_tensor[idx].cuda()

The library does this by masquerading the pinned-memory CPU tensor as a cupy GPU tensor and using the cupy GPU indexing kernels. It’s not really speeding up the CPU-GPU copy part, it’s avoiding the overhead of a separate CPU indexing operation. (The GitHub page mentions “memmaps”, but it’s not using memory mapping at all).
For CPU <-> GPU copies, the relative speed-up depends on how slow the PyTorch CPU indexing operation is. For the (131072, 128) size, this can be as fast as 2 ms on a multi-core machine or ~25 ms on collab notebook with only one or two virtual CPUs.  For comparison, the CPU-GPU copy from pinned memory is consistent at ~5-6 ms. (Again, it’s not the CPU-GPU copy that’s sped up; it’s the separate CPU indexing operation that’s avoided)
There’s also a significant indexing performance bug in PyTorch 1.1 and 1.2 (fixed in the nightly builds) that makes the indexing much slower. The apparent GPU <-> GPU indexing speed-ups are entirely due to this bug. You could either use the nightly builds or use index_select / index_copy_ instead of a[idx] notation in 1.1/1.2 to avoid that slow down in vanilla PyTorch.  (Bug is described in #24083 8 for reference)
The library is missing some synchronization. Particularly, when copying from GPU to pinned memory (masquerading as GPU via cupy), you need to synchronize before accessing the CPU data; otherwise it may not be consistent.
There’s a few bugs in the benchmark code, mostly minor:

sampl = np.random.uniform(low=-1.0, high=1.0, size=(1000000, 128)). This is float64; it should be float32 to match everything else:
sampl = np.random.uniform(low=-1.0, high=1.0, size=(1000000, 128)).astype(np.float32)

The timing code should have a torch.cuda.synchronize() at the end. Instead of:

torch.cuda.synchronize()
cupy.cuda.Device().synchronize()

runningTime=0
for i in range(numSamples):
    start = time.time()

    with torch.no_grad():

        gadgetCPU.insertData(u_embeddings.weight.data, indexess )

    end = time.time()
    runningTime = runningTime + end - start

print('set corpus. cupy live pinned')
print(runningTime/numSamples)

It should be written as:
torch.cuda.synchronize()
cupy.cuda.Device().synchronize()

start = time.time()
for i in range(numSamples):
    with torch.no_grad():
        gadgetCPU.insertData(u_embeddings.weight.data, indexess )

torch.cuda.synchronize()
end = time.time()
runningTime = end - start

print('set corpus. cupy live pinned')
print(runningTime/numSamples)


Question: Ask and answer technical questions here. Say hello to the community!

Context: Hey. I wanna start tomorrow the exam for the TensorFlow developer certificate. It is tensorflow 2.5 required. I have a Mac with M1 and only tensorflow-macos runs on the machine. Does somebody knows if it is still possible to use tensorflow-macos with installing the plugin or is it only possible to use tensorflow 2.5 . If I can use only tensorflow 2.5 somebody knows a solution how to run this version on an M1? Thank you.


Problem on connecting tensorflow with cuda

Question:
Hii, i want to install tensorflow in a linux machine where cuda is present.
±----------------------------------------------------------------------------+
| NVIDIA-SMI 390.77                 Driver Version: 390.77                    |
|-------------------------------±---------------------±---------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K40c          On   | 00000000:03:00.0 Off |                    0 |
| 24%   45C    P8    23W / 235W |     11MiB / 11441MiB |      0%      Default |
±------------------------------±---------------------±---------------------+
|   1  Quadro K620         On   | 00000000:04:00.0  On |                  N/A |
| 36%   49C    P8     1W /  30W |    573MiB /  1999MiB |      0%      Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    1      1781      G   /usr/bin/X                                    22MiB |
|    1     58011      G   /usr/bin/X                                    34MiB |
|    1     59222      G   /usr/bin/gnome-shell                          87MiB |
|    1     66073      G   /usr/bin/gnome-shell                         186MiB |
|    1    110408      G   /usr/bin/X                                   126MiB |
|    1    111855      G   /usr/bin/gnome-shell                         101MiB |
±----------------------------------------------------------------------------+
but when i import import tensorflow as tf it gives some kind of error
2022-01-28 15:16:56.056131: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library ‘libcudart.so.11.0’; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-9.1/lib64
2022-01-28 15:16:56.056200: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

Context: Can you run nvcc --version? What is the output?


The different parameter size of mobilenetv3 in 2.7.0 from nightly

Question:
Hello, I was training a model using Mobilenet V3 small.
The number of parameters of MobileNetv3Small(alpha=1) in tf=2.7.0 is 1.5M, while it is 0.9M in tf-nightly
The same code was used to load the MobileNetV3Small model.
base_model =keras.applications.MobileNetV3Small(input_shape=(384, 512, 3),include_top=False,weights='imagenet',)  base_model.summary()
I saw that some layers were deleted in tf-nightly. What is that? Which one is the correct one?

Context:
Some layers was moved with this commit:

      github.com/keras-team/keras

      Update MobileNetV3 architecture, move some layers from body to head, which only appears when `include_top=True`.

        committed

        Oct 9, 2021

          haifeng-jin

          +52
          -27

    Pretrained weights are updated and tested.

PiperOrigin-RevId: 401845877

So the weights hash changed


R2.2 libtensorflowlite_c.so crash for multithread

Question:
C35033A  01-04 05:29:57.710   498 12173 F libc    : /buildbot/src/android/ndk-release-r18/external/libcxx/…/…/external/libcxxabi/src/abort_message.cpp:73: abort_message: assertion “cannot create thread specific key for __cxa_get_globals()” failed
C350342  01-04 05:29:57.712   498 12173 F libc    : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 12173 (Blur), pid 498 (provider@2.4-se)
C350630  01-04 05:29:57.860 12179 12179 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
C350631  01-04 05:29:57.860 12179 12179 F DEBUG   : Native Crash TIME: 16201049
C350632  01-04 05:29:57.860 12179 12179 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
C350633  01-04 05:29:57.860 12179 12179 F DEBUG   : Build fingerprint: ‘realme/RMP2105/RE87CCL1:11/RP1A.201005.001/1640606145:userdebug/test-keys’
C350634  01-04 05:29:57.861 12179 12179 F DEBUG   : Revision: ‘0’
C350635  01-04 05:29:57.861 12179 12179 F DEBUG   : ABI: ‘arm’
C350636  01-04 05:29:57.861 12179 12179 F DEBUG   : Timestamp: 2022-01-04 05:29:57+0800
C350637  01-04 05:29:57.861 12179 12179 F DEBUG   : pid: 498, tid: 12173, name: Blur  >>> /vendor/bin/hw/android.hardware.camera.provider@2.4-service <<<
C350638  01-04 05:29:57.861 12179 12179 F DEBUG   : uid: 1047
C350639  01-04 05:29:57.861 12179 12179 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
C35063A  01-04 05:29:57.861 12179 12179 F DEBUG   : Abort message: ‘/buildbot/src/android/ndk-release-r18/external/libcxx/…/…/external/libcxxabi/src/abort_message.cpp:73: abort_message: assertion “cannot create thread specific key for __cxa_get_globals()” failed’
C35063B  01-04 05:29:57.861 12179 12179 F DEBUG   :     r0  00000000  r1  00002f8d  r2  00000006  r3  cc06d010
C35063C  01-04 05:29:57.861 12179 12179 F DEBUG   :     r4  cc06d024  r5  cc06d008  r6  000001f2  r7  0000016b
C35063D  01-04 05:29:57.861 12179 12179 F DEBUG   :     r8  cc06d010  r9  cc06d020  r10 cc06d040  r11 cc06d030
C35063E  01-04 05:29:57.861 12179 12179 F DEBUG   :     ip  00002f8d  sp  cc06cfe0  lr  ee9087cd  pc  ee9087e0
C35064A  01-04 05:29:57.867 12179 12179 F DEBUG   : backtrace:
C35064B  01-04 05:29:57.867 12179 12179 F DEBUG   :       #00 pc 000387e0  /apex/com.android.runtime/lib/bionic/libc.so (abort+172) (BuildId: 724f04e3eb055a58d7517ffbc7210561)
C35064C  01-04 05:29:57.867 12179 12179 F DEBUG   :       #01 pc 00038a87  /apex/com.android.runtime/lib/bionic/libc.so (__assert2+22) (BuildId: 724f04e3eb055a58d7517ffbc7210561)
C35064D  01-04 05:29:57.867 12179 12179 F DEBUG   :       #02 pc 001fde95  /vendor/lib/libtensorflowlite_c.so
C35064E  01-04 05:29:57.867 12179 12179 F DEBUG   :       #03 pc 001fc471  /vendor/lib/libtensorflowlite_c.so
C35064F  01-04 05:29:57.867 12179 12179 F DEBUG   :       #04 pc 00081cb5  /apex/com.android.runtime/lib/bionic/libc.so (pthread_once+76) (BuildId: 724f04e3eb055a58d7517ffbc7210561)
C350650  01-04 05:29:57.867 12179 12179 F DEBUG   :       #05 pc 001fc433  /vendor/lib/libtensorflowlite_c.so
C350651  01-04 05:29:57.867 12179 12179 F DEBUG   :       #06 pc 001fc3e1  /vendor/lib/libtensorflowlite_c.so
C350652  01-04 05:29:57.867 12179 12179 F DEBUG   :       #07 pc 001fc15b  /vendor/lib/libtensorflowlite_c.so
C350653  01-04 05:29:57.867 12179 12179 F DEBUG   :       #08 pc 001fbf05  /vendor/lib/libtensorflowlite_c.so
C350654  01-04 05:29:57.867 12179 12179 F DEBUG   :       #09 pc 001fbfe5  /vendor/lib/libtensorflowlite_c.so
C350655  01-04 05:29:57.867 12179 12179 F DEBUG   :       #10 pc 001fbfa3  /vendor/lib/libtensorflowlite_c.so
C350656  01-04 05:29:57.867 12179 12179 F DEBUG   :       #11 pc 000fd855  /vendor/lib/libtensorflowlite_c.so
C350657  01-04 05:29:57.867 12179 12179 F DEBUG   :       #12 pc 00080973  /apex/com.android.runtime/lib/bionic/libc.so (__pthread_start(void*)+40) (BuildId: 724f04e3eb055a58d7517ffbc7210561)
C350658  01-04 05:29:57.867 12179 12179 F DEBUG   :       #13 pc 00039ce3  /apex/com.android.runtime/lib/bionic/libc.so (__start_thread+30) (BuildId: 724f04e3eb055a58d7517ffbc7210561)

Context:
Are you using multiple interpreter instances?

      TensorFlow

tflite::Interpreter Class Reference  |  TensorFlow Lite

WARNING: This class is not thread-safe. The client is responsible for ensuring serialized interaction to avoid data races and undefined behavior.


Question:
Hi there,
I’d like to know which tensorflow-datasets versions are compatible with which versions of tensorflow? I’m using tensorflow together with autokeras on GPU, it took some time to find matching versions that would recognize the GPU. Now I installed tensorflow-datasets, autokeras doesn’t work anymore. It would be helpful if someone could point me to a list of which TFDF versions go with which TF versions so I can install specific versions in my conda env.

Context:
Hi @Julia_Wasala Welcome to the TF Forum

 Julia_Wasala:

I’m using tensorflow together with autokeras on GPU, it took some time to find matching versions that would recognize the GPU.

Can you please share your versions of CUDA, cuDNN, GPU driver, etc (دعم GPU  |  TensorFlow)

 Julia_Wasala:

Now I installed tensorflow-datasets, autokeras doesn’t work anymore.

Will try to chase the right team to look into this. Which version of Autokeras, TFDS and TensorFlow do you have right now?
As far as I understand the min version of TF for TFDS is 2.1: datasets/tf_compat.py at 93c1d96dee42b451038e9303166e4c7e83a26b82 · tensorflow/datasets · GitHub


AttributeError: ‘KerasTensor’ object has no attribute ‘node’

Question:
How to get rid of this:
AttributeError: 'KerasTensor' object has no attribute 'node'
Was working fine with TF 2.4.1 but giving problem with TF 2.7.0
Script
types_inp = Input(shape=(self.len_of_types, ), dtype='int32')
.......
out = Dense(self.num_ways, activation='softmax')(prob)
model = keras.Model(inputs=[types_inp,..],outputs=out]

Error:
AttributeError: 'KerasTensor' object has no attribute 'node'

Context:
I don’t know if you could minimize a share few lines of standalone code to reproduce this (or a Colab).
In the meantime you could check if it could be related to:

      github.com/tensorflow/tensorflow

      [TF 2.4] KerasTensor breaks typing compatibility

        opened

        Nov 5, 2020

          lgeiger

          type:bug

          comp:keras

          TF 2.4

    **System information**
- Have I written custom code (as opposed to using a stoc…k example script provided in TensorFlow): yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS
- TensorFlow installed from (source or binary): binary
- TensorFlow version (use command below): v2.4.0-rc0 and tf-nightly
- Python version: 3.8

## The Problem
Since version 2.4 functional Keras models use `KerasTensor` instead of `tf.Tensor` as layer output types. Unfortunately `KerasTensor` doesn't subclass `tf.Tensor` which breaks `isinstance(x, tf.Tensor)` checks:
https://github.com/tensorflow/tensorflow/blob/6d9e0887f6bce8fbeeb430364e520d05350d96d5/tensorflow/python/keras/engine/keras_tensor.py#L63

The release notes recommend to use `tf.is_tensor` instead.
In my opinion this is not really Pythonic and breaks compatibility with the [TF Types RFC](https://github.com/tensorflow/community/blob/master/rfcs/20200211-tf-types.md) (/cc @mdanatg) wich even mentiones that `tf.is_tensor` is expected to be deprecated eventually.

Concretely, switching from `isinstance(x, tf.Tensor)` to `tf.is_tensor` is also not possible in all cases. E.g. this breaks usage of static typecheckers like `pytype` or `typeguard`:

A common pattern which can also be found in TensorFlow Addons (/cc @seanpmorgan) is the following:

```python
from typeguard import typechecked
import tensorflow as tf

@typechecked
def foo(x: tf.Tensor):
    print(x.dtype)

foo(tf.keras.Input(shape=(32, 32, 3)))  # Throws in TF 2.4 since `isintance` is used for typechecking
```

## Possible solutions

1. Make `KerasTensor` a subclass of `tf.Tensor`. @mihaimaruseac @fchollet is there a reason why this isn't currently the case?

2. Make `KerasTensor` a subclass of [`types.Tensor`](https://github.com/tensorflow/tensorflow/blob/6d9e0887f6bce8fbeeb430364e520d05350d96d5/tensorflow/python/types/core.py#L40-L54). I don't see any disadvantage of doing this in general, but it wouldn't really fix this issue since `types.Tensor` is not exposed as part of the public API yet so users would need to rely on private TensorFlow APIs

3. In usercode this could be fixed by directly relying on `KerasTensor` to replace the usage of `tf.Tensor` with:
   ```python
   from typing import Union
   from tensorflow.python.keras.engine.keras_tensor import KerasTensor

   TensorType = Union[tf.Tensor, KerasTensor]
   ```
   I do not think this is a proper solution since it requires users to rely on internal APIs and implementation details that might change in the future.

I am currious to hear back from you on what the bestpractices for type checking of Tensors are, or whether I am just missing somthing obvious here.


Question:
Hello everyone,
I want to reproduce this example 2, but I would like the input_data to be a Text Nested List instead. In other words:
text_dataset = tf.data.Dataset.from_tensor_slices(["foo", "bar", "baz"])
max_features = 5000
max_len = 4

# -------- EXPERIMENTAL ADDENDUM -----------------
def split_on_comma(input_data):
    return tf.strings.split(input_data, sep=", ")
# ------------------------------------------------

vectorize_layer = tf.keras.layers.TextVectorization(
    max_tokens=max_features,
    output_mode='int',
    output_sequence_length=max_len,
    split=split_on_comma
)

with tf.device("/CPU:0"):
    vectorize_layer.adapt(text_dataset.batch(64))

inputs = tf.keras.Input(shape=(1,), dtype=tf.string)
x = vectorize_layer(inputs)
model = tf.keras.Model(inputs=inputs, outputs=x)

# input_data = [["foo qux bar"], ["qux baz"]] # Original input, I DO NOT want this...
input_data = [["foo", "qux", "bar"], ["qux baz"]] # ...I WANT THIS AS INPUT INSTEAD
input_data = [[', '.join(x)] for x in input_data] # MY EXPERIMENTAL WORKAROUND
model.predict(input_data)

The thing is that I have a dataset where one of the features consists of a list of strings, with different length each, that do not share any context with each other, and I want to use it as training input for some Model with the Functional  API. Splitting it in different columns (OHE) is out of question, as this feature is highly sparse (~7K different classes).
I am a bit concerned that maybe my implementation is not the best neither correct, because I am getting different results; for instance, this is the example’s original output:
array([[2,1,4,0],
       [1,3,0,0]])

And what I am getting is:
array([[1,0,0,0],
       [1,0,0,0]])

Finally, I am not quite sure how (if?) Ragged Tensors would be of use with TextVectorization layer (again, the goal is building a Model using the Functional API). I have found some examples, but not exactly focused towards TextVectorization. Plus, I get that TextVectorization can process “one string per sample”, but if so, what would be other suggestions to what I am aiming here?
Have someone have had this problem before? How did you tackled it?
Thank you very much.

Context:
array([[1,0,0,0],
       [1,0,0,0]])

Somehow your split is failing to split and it’s all coming in as one token. The zeros are padding, the ones are probably <UNKNOWN>. I looked at this a bit, and it’s because the default standardize is “lower and strip punctuation” → it’s removing the commas.

I am not quite sure how (if?) Ragged Tensors would be of use with TextVectorization layer

Ragged Tensors are exactly what you’re looking for here. Your split function returns ragged tensors.
If you split before passing to the TextVectorization layer (the result will be ragged), just tell the TextVectorization layer not to split by passing split=None.

the goal is building a Model using the Functional API

If you have a ragged input to a functional model use keras.Input(..., ragged=True).
If you’re not using any other features of TextVectorization then StringLookup may be a good alternative since it doesn’t have all the bells and whistles.


On-device-training using tflite_runtime wheel built based tf2.7

Question:
Hello,
I’m trying to explore the possibilities of launching on device training on a raspberry 3B+ using the tflite_runtime python wheel that I have built manually based on the release 2.7 which says it supports on-device-training. I’m following currently the examples provided by these links:
On-Device Training with TensorFlow Lite
On-device training in TensorFlow Lite — The TensorFlow Blog
The Python examples starts by importing tensorflow which is not available on my board, I was wondering if there is a way to implement the same example but with the tflite_runtime.
Any help or examples would be appreciated.
Thank you in advance,

Context:
The on-device part start from “Convert model to TensorFlow Lite format” section in:

      TensorFlow

On-Device Training with TensorFlow Lite 1


How can I do any constraints in the same layer?

Question:
Dears,
I want to do two weights constraints in the same layer. My weights should be non-negatives and w_1 > w_2 > w_3 > … > w_N. I would like to use tf.keras.constraints.  Did you do this?
Regards.

Context:
If you check the other “canned” constrains you can see that you need to just manipulate the weights that are not respecting your constrain.
In your case I suggest to check this implementation as a baseline to retrieve the indexes of the weights that you want to manipulate:

      github.com

    tensorflow/tensorflow/blob/v2.7.0/tensorflow/python/ops/check_ops.py#L2108-L2148

          @tf_export(
              'math.is_strictly_increasing',
              v1=[
                  'math.is_strictly_increasing', 'debugging.is_strictly_increasing',
                  'is_strictly_increasing'
              ])
          @dispatch.add_dispatch_support
          @deprecation.deprecated_endpoints('debugging.is_strictly_increasing',
                                            'is_strictly_increasing')
          def is_strictly_increasing(x, name=None):
            """Returns `True` if `x` is strictly increasing.

  Elements of `x` are compared in row-major order.  The tensor `[x[0],...]`
            is strictly increasing if for every adjacent pair we have `x[i] < x[i+1]`.
            If `x` has less than two elements, it is trivially strictly increasing.

  See also:  `is_non_decreasing`

  >>> x1 = tf.constant([1.0, 2.0, 3.0])
            >>> tf.math.is_strictly_increasing(x1)

  This file has been truncated. show original


Converting tensorflow model on ssdlite model

Question:
Hello,
i’m on personal project and i’m trying to detect custom object with openCV
i have already use the detectmultiiscale function with a haarcascade model but results are just under my satisfication
So i’m trying to use tensorflow and with a code that i found on internet (and it works):
while(True):

    t1 = cv2.getTickCount()

    ret, frame = camera.read()
    frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    frame_expanded = np.expand_dims(frame_rgb, axis=0)

    (boxes, scores, classes, num) = sess.run(
        [detection_boxes, detection_scores, detection_classes, num_detections],
        feed_dict={image_tensor: frame_expanded})

    vis_util.visualize_boxes_and_labels_on_image_array(
        frame,
        np.squeeze(boxes),
        np.squeeze(classes).astype(np.int32),
        np.squeeze(scores),
        category_index,
        use_normalized_coordinates=True,
        line_thickness=8,
        min_score_thresh=0.85)

    if cv2.waitKey(1) == ord('q'):
        break

it works with a SSDLITE model now i have to create my own custom model
So my question is : How to train an SSDLITE model ? and is that the best way for object detection ?

Context:
Partially we have talked about this at:

    Tensorflow HUB - transfer learning for object detection General Discussion

    Hi all,
I am having some trouble with applying transfer learning in object detection models.
The documentation provides a nice tutorial for transfer learning in classification models (Retraining an Image Classifier  |  TensorFlow Hub). However, I couldn’t find anything similar for the object detection case.
I tried to readapt the object detection tutorial (TensorFlow Hub Object Detection Colab) according to the classification tutorial linked above, but I didn’t manage to progress much and got…


How to reduce Initialized session time when to use GPU

Question:
Using the benchmark tool to analyze tflite model run time and memory, found that when setting use_gpu=true,  "initialized session" time is very long,while setting false the time is short.
   so i want to know  how to reduce Initialized session time when to use_gpu? thanks

Context:
Do you have any code snippet of what you tried?
are you using the cmd line tool?


What model is “Taking Advantage of Context Features” tutorials based on?

Question:
Hello,
i have a questions regarding this tutorial:

      TensorFlow

Taking advantage of context features  |  TensorFlow Recommenders 3

what recommendation model does this tutorial based on? is it Collaborative Filtering?
and if so why does it not use the ratings in the dataset used?
how did the model come up with the recommendation given, without the rating?

Context: @Wei_Wei may help here


What is the good way to access post-transform-statistics in Trainer (TFX)

Question:
Hi, all.
I have a question about TFX’s Trainer component.
How can I access post-transform-statistics(output of Transform component) in the Trainer’s run_fn?
I checked that post-transform-statistics is emitted by Transform, but standard_artifacts.ExampleStatistics cannot be passed to Trainer

tfx/standard_component_specs.py at v1.5.0 · tensorflow/tfx · GitHub 1 (TransformSpec)

tfx/standard_component_specs.py at v1.5.0 · tensorflow/tfx · GitHub 1 (TrainerSpec)

I know I can parse version in the path and join paths for the post-transform-statistics using fn_args.transform_graph_path that is the argument of Trainer’s run_fn, but I don’t think that is a good option.
Thanks

Context:
Have you tried using a custom_config argument for Trainer?  Something like:
trainer = tfx.components.Trainer(
    ...
    custom_config={
        'tft_stats':transform.outputs['post_transform_stats']
        }
)


Visualizing the custom components’ results in the Pipeline UI using TFX KubeflowDAGRunner

Question:
Hi,
I have a question about visualizing results in the KubeFlow’s Pipeline UI using TFX KubeflowDAGRunner.
Currently, after each component ended, TFX writes the results into the mlpipeline_ui_metadata.json, regardless of whether this file already exists.
(_dump_ui_metadata function in the entry point of KubeflowDAGRunner: tfx/container_entrypoint.py at 278ec3bd300f41617a980ab06f11402124edb874 · tensorflow/tfx · GitHub)
The default visualizations are very helpful, but I want to extend them for my custom components’ results.
Is there any way to visualize custom components’ results in the Pipeline UI?
Related issue:

      github.com/tensorflow/tfx

      Customizing `mlpipeline_ui_metadata.json` 1

        opened

        May 13, 2021

          jeongukjae

          type:feature

          stat:awaiting tensorflower

    **System information**

- TFX Version (you are using): 0.29
- Environment in …which you plan to use the feature (e.g., Local
  (Linux/MacOS/Windows), Interactive Notebook, Google Cloud, etc..): Google Cloud
- Are you willing to contribute it (Yes/No): Yes

**Describe the feature and the current behavior/state.**

https://github.com/tensorflow/tfx/blob/79fa0d8adce177d04feaa737f1440689344521df/tfx/orchestration/kubeflow/container_entrypoint.py#L210-L296

For now, `_dump_ui_metadata` function in `tfx/orchestration/kubeflow/container_entrypoint.py` does not consider already existing `/mlpipeline-ui-metadata.json`. So if I write some metadata that I want to see in KubeFlow UI in a custom component, the file will be overwritten.

If `/mlpipeline-ui-metadata.json` already exists(maybe created in the component) and has a valid format, I think extending an existing file is better.

**Will this change the current API? How?**

No

**Who will benefit with this feature?**

**Do you have a workaround or are completely blocked by this?** :

**Name of your Organization (Optional)**

**Any Other info.**

Thanks

Context:
Thank you for the question. I think that we need to extend TFX to accommodate user generated outputs as you suggested.
Custom components can generate their visualization according to the kubeflow visualization guide 1 and KubeflowDagRunner can generate common ones and merge them into the final output.


Question:
Hi All,
New to TensorFlow and Neural Networks in general, apologies if this is an overly simple question. I did do some digging but didn’t find what I am looking for. I’m wanting to build a model that accepts different types of data as multiple inputs. For example, a US based, 5 digit, postal code and the time of day, expressed as an integer (1 - morning, 2 - afternoon, 3 - evening, etc.).
I’m not sure if it’s the right approach but, I’m one hot encoding the data. Postal codes in this case look like (note, I’m working in JS):
tf.tensor3d([
	[
		// 94701
		[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 1 ],
		[ 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 ],
		[ 0, 0, 0, 0, 0, 0, 0, 1, 0, 0 ],
		[ 1, 0, 0, 0, 0, 0, 0, 0, 0, 0 ],
		[ 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ]
	],
	...
]);

This has worked really well as a single input, struggling to introduce the second one however. Data for the time of day looks like:
tf.tensor3d([
	[
		// evening
		[ 0, 0, 1, 0 ]
	],
	...
]);

Everything I’ve tried so far leads me to shape mismatch errors. I’ve tried concatenating the tensors across different axis, tried concatenating layers, and have tried using an array of inputs with tf.model().
Seems like multiple, different data type, inputs should be a common use case. Am I thinking about this wrong? Should I be trying to normalize the data so it takes the same shape (use 10 digits to represent the time of day for example)? Should I be combining the data into a single tensor? Something else I’m missing?
Many thanks!

Context:
Welcome and thanks for the question certainly a great one!
How come you are using tensor3d here? Maybe I am misunderstanding what you are trying to do, but if you just have a single post code and a single time of day as input to the model I would have expected it to be in the form of:
// 94701                        // evening
[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0 ]

So the above would be a tensor1d which when sent as a batch to the model would be a tensor2d (can just call expandDims() to do that.
So as a batch you can do multiple predictions which would look like:
// Tensor2D
[
  [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0 ],
  [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0 ],
  [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0 ],
  [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0 ]
]

You do not need to normalize 1 hot encoded data here.
PS if you are just starting out you may be interested in my upcoming course for TensorFlow.js that covers some topics like this over on edX. Free to do if you do not need certification:

      edX

Google AI for JavaScript Developers with TensorFlow.js 1

  Get productive with TensorFlow.js - Google's Machine Learning library for JavaScript. From pre-made off the shelf models to writing or training your own, learn how to create next gen web apps.


Question:
Hello communit,
i saved a model which i trained for a deep-q learning project. Tensorflow/keras 2.7.0 on Python 3.9.9 64bit on Windows 10. When i load the model again and use it for inference, it works fine.
It is not a very special model:
        model = Sequential()
        model.add(Conv2D(256, (3, 3), input_shape=self.env.OBSERVATION_SPACE_VALUES))
        model.add(Activation('relu'))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.2))
        model.add(Conv2D(256, (3, 3)))
        model.add(Activation('relu'))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.2))
        model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
        model.add(Dense(64))
        model.add(Dense(self.env.ACTION_SPACE_SIZE, activation='linear'))
        model.compile(loss="mse", optimizer=Adam(learning_rate=LEARNING_RATE), metrics=['accuracy'])

Savig
model.save(model_filename)

Loading
model = keras.models.load_model(model_filename)

I want to use the model on a tf/keras 2.7.0 setup on a Raspberry Pi OS 64bit (bullseye) machine. When i use the same code for loading the model as on the Windows machine, i get the following error:
/usr/local/lib/python3.9/dist-packages/keras/saving/saved_model/load.py:116: RuntimeWarning: Unexpected end-group tag: Not all data was converted
  metadata.ParseFromString(file_content)

What does this mean?

Context: Did you convert the model to TFLite before trying to load it on Raspberry Pi?


How to handle null images (no class) using CSV format for Model Maker

Question:
Loading via object_detector.DataLoader.from_csv(filename) using the CSV format.

      Google Cloud

Formatting a training data CSV  |  AutoML Vision Object Detection... 1

I just cant figure out how to provide no-class (null) examples via CSV.
Pascal VOC seems to support that but I put a lot of effort into a Notebook and realized my null images are lost along the way.
Create a fake class with no bounding box? Doesn’t feel correct.

Context:
Trying some alternatives.
Modified one row of my csv-dataset to have an empty class label and set all bounding box vertices columns to string to allow passing an empty string.
  # TODO: Remove this since it's only here to mockup a row where trying to insert a null-class example
  #       This might fail since documentation suggest this is not possible
  mycsv['xmin'] = mycsv['xmin'].astype(str)
  mycsv['ymin'] = mycsv['ymin'].astype(str)
  mycsv['xmax'] = mycsv['xmax'].astype(str)
  mycsv['ymax'] = mycsv['ymax'].astype(str)
  mycsv.at[0,'class'] = ''
  mycsv.at[0,'xmin'] = ''
  mycsv.at[0,'ymin'] = ''
  mycsv.at[0,'xmax'] = ''
  mycsv.at[0,'ymax'] = ''

Fails during load:
/usr/local/lib/python3.7/dist-packages/tensorflow_examples/lite/model_maker/core/data_util/object_detector_dataloader_util.py in _get_xml_dict_from_csv_lines(images_dir, image_filename, lines)
    335   for line in lines:
    336     label = line[2].strip()
--> 337     xmin, ymin = float(line[3]) * width, float(line[4]) * height
    338     xmax, ymax = float(line[7]) * width, float(line[8]) * height
    339     obj = {

ValueError: could not convert string to float:

Leaving bbox columns as float and inserting None yeilds the same result
  mycsv.at[0,'class'] = ''
  mycsv.at[0,'xmin'] = None
  mycsv.at[0,'ymin'] = None
  mycsv.at[0,'xmax'] = None
  mycsv.at[0,'ymax'] = None

Columns xmin, xmax, ymin and ymax becomes strings:

Expand Table

split
filename
class
xmin
ymin
empty1
empty2
xmax
ymax
empty3
empty4

TEST
file1.jpg

TEST
file2.jpg
red
0.27
0.0

1.0
1.0

TEST
file3.jpg
blue
0.37
0.7866666666666666

1.0
1.0

TEST
file4.jpg
blue
0.426
0.18

0.67
0.59

Fails during load again:
/usr/local/lib/python3.7/dist-packages/tensorflow_examples/lite/model_maker/core/data_util/object_detector_dataloader_util.py in _get_xml_dict_from_csv_lines(images_dir, image_filename, lines)
    335   for line in lines:
    336     label = line[2].strip()
--> 337     xmin, ymin = float(line[3]) * width, float(line[4]) * height
    338     xmax, ymax = float(line[7]) * width, float(line[8]) * height
    339     obj = {

ValueError: could not convert string to float:


Question:
I’ve installed the cuda toolkit and cudnn on my Ubuntu machine to run my models on the GPU. My LD_LIBRARY_PATH is set to the absolute path, “/usr/local/cuda/include:/usr/local/cuda/lib64.” If I were to run the following script as a .py file, tensorflow can see my GPU.
import tensorflow as tf

tf.config.list_physical_devices('GPU')

The result is [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')].
However, if I were to run the same code block on a Jupyter Notebook, the returned list is empty. I am unsure as to why the notebook cannot see my GPU. I have tried setting its own environment variable to the same value as my $LD_LIBRARY_PATH environment variable, but no such luck.

Context:
Can you try to test your Notebook with a prepared Docker environment with the official Jupyter gpu image:

      TensorFlow

Docker  |  TensorFlow 1


An Example of few shot learning for image classification (not detection)?

Question:
I was trying to get my hands on few shots learning but for image classification, however all the samples i get are of image detection
i was wondering how mainstream is few shot learning for image classification like that of MNIST
Can someone point me towards the sample code of the same ?
All the help is appreciated

Context:
I think you could find something similar in Omniglot:

      keras.io

Keras documentation: Few-Shot learning with Reptile

      Medium – 30 Jul 20

Re-implementation of the Prototypical Network for Few-Shot Learning using...

  This article is about the implementation based on the paper Prototypical Networks for Few-shot Learning (NIPS 2017)

    Reading time: 7 min read


Question:
I’m pretty new so sorry if my question is elementary.
I am trying to basally replicate the Fashion MNIST tutorial but with my own images. My own images are only in two classes: ‘candle’ and ‘no candle’.
I load my data using the following:
ds_train = tf.keras.preprocessing.image_dataset_from_directory(
‘C:/Users/Eric/Desktop/Candle Data’,
labels=‘inferred’,
label_mode=“int”,
color_mode=‘grayscale’,
batch_size=4,
image_size=(256, 256),
shuffle=True,
seed=123,
)
I think that this is an ok as far as how to load my images although I am not certain.
The image files are on my computer in two folders named: ‘Candle’ and ‘No Candle’ at the location above.
In the tutorial they define the following:
train_images and train_labels
Then the images and labels are used to make the model:
model.fit(train_images, train_labels, epochs=10)
I think that I have loaded the images to ‘ds_train’ but I don’t have the labels. When I created ‘ds_train’ I used the ‘infered’ labels but I don’t know if that created the necessary labels or not.
How should I define the labels for my images, or have I done this already?
Thank you for any help

Context:
You can read what each of the parameters means here: tf.keras.utils.image_dataset_from_directory  |  TensorFlow Core v2.7.0
At present, the algorithm in your code assigns integer class labels to images that it finds in the nested folders of the root folder with data.
Both images and labels are stored in the Dataset object (ds_train), which you can pass to model.fit()


Question:
i’m facing this problem
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
     (0) Invalid argument:  In[0] mismatch In[1] shape: 1108 vs. 1120: [42,1108] [1120,256] 0 0
   [[node model/dense/MatMul (defined at rnn_flickr_fit.py:273) ]]
   (1) Invalid argument:  In[0] mismatch In[1] shape: 1108 vs. 1120: [42,1108] [1120,256] 0 0
 [[node model/dense/MatMul (defined at rnn_flickr_fit.py:273) ]]
  [[Adam/Cast_6/ReadVariableOp/_6]]
   0 successful operations.
    0 derived errors ignored. [Op:__inference_train_function_7431]
 Function call stack:
    train_function -> train_function
   2021-07-02 14:23:52.766840: W   tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred   when finalizing GeneratorDataset iterator: Failed precondition: Python   interpreter state is not initialized. The process may be terminated.
    	 [[{{node PyFunc}}]]
what should i check to solve it ?

Context: Can you add a minimal code snippet that reproduce this issue please?


List index out of range while saving a trained model

Question:
PLEASE HELP !!
I’m trying to fine-tune a pre-trained DistilBERT model from Huggingface using Tensorflow. Everything runs smoothly and the model builds and trains without error. But when I try to save the model it stops with the error “IndexError: list index out of range”. I’m using pycharm with TPU.
Any help would be much appreciated!
Code:
import h5py
import numpy as np
import pandas as pd
import pydot
import simplejson as simplejson
import tensorflow as tf
import os
from transformers import pipeline
from tensorflow import keras
train = pd.read_csv("train.csv")
print("Training_dataset_shape:", train.shape)

MAX_LEN=100

from transformers import BertTokenizer, TFBertModel, TFAutoModel,AutoTokenizer

#model_name = "bert-base-multilingual-cased"
#tokenizer = BertTokenizer.from_pretrained(model_name) # FC: this is the tokenizer we will use on our text data to tokenize it

model_name = "huggingface/distilbert-base-uncased-finetuned-mnli"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# FC we make a function in order to have a list of the id for each word and the separator
def encode_sentence(s):
   tokens = list(tokenizer.tokenize(s)) # FC: split the sentence into tokens that are either words or sub-words
   tokens.append('[SEP]') # FC: a token called [SEP] (=separator) is added to mark end of each sentence
   return tokenizer.convert_tokens_to_ids(tokens) # # FC: instead of returning the list of tokens, a list of each token ID is returned

def bert_encode(hypotheses, premises,
                tokenizer):  # FC: for RoBERTa we remove the input_type_ids from the inputs of the model

    num_examples = len(hypotheses)

    sentence1 = tf.ragged.constant([  # FC: constructs a constant ragged tensor. every entry has a different length
        encode_sentence(s) for s in np.array(hypotheses)])

    sentence2 = tf.ragged.constant([
        encode_sentence(s) for s in np.array(premises)])

    cls = [tokenizer.convert_tokens_to_ids(['[CLS]'])] * sentence1.shape[
        0]  # FC: list of IDs for the token '[CLS]' to denote each beginning

    input_word_ids = tf.concat([cls, sentence1, sentence2],
                               axis=-1)  # FC: put everything together. every row still has a different length.

    # input_word_ids2 = tf.concat([cls, sentence2, sentence1], axis=-1)

    # input_word_ids = tf.concat([input_word_ids1, input_word_ids2], axis=0) # we duplicate the dataset inverting sentence 1 and 2

    input_mask = tf.ones_like(
        input_word_ids).to_tensor()  # FC: first, a tensor with just ones in it is constructed in the same size as input_word_ids. Then, by applying to_tensor the ends of each row are padded with zeros to give every row the same length
    type_cls = tf.zeros_like(cls)
    type_s1 = tf.zeros_like(sentence1)
    type_s2 = tf.ones_like(sentence2)
    input_type_ids = tf.concat(
        [type_cls, type_s1, type_s2], axis=-1).to_tensor()  # FC: concatenates everything and again adds padding
    inputs = {
        'input_word_ids': input_word_ids.to_tensor(),  # FC: input_word_ids hasn't been padded yet - do it here now
        'input_mask': input_mask}

    return inputs

train_input = bert_encode(train.premise.values, train.hypothesis.values, tokenizer)
# total_train_input = bert_encode(total_train.premise.values, total_train.hypothesis.values, tokenizer)

max_len = 136  #: FC 50 in the initial tutorial

def build_model():
    encoder = TFAutoModel.from_pretrained(model_name)
    input_word_ids = tf.keras.Input(shape(max_len,),dtype=tf.int32,name="input_word_ids")
    input_mask = tf.keras.Input(shape=(max_len,), dtype=tf.int32,name="input_mask")
    embedding = encoder([input_word_ids, input_mask])[0]  # FC: add_input_type_ids for the BERT model
    output = tf.keras.layers.Dense(3, activation='softmax')(embedding[:, 0, :])
    model = tf.keras.Model(inputs=[input_word_ids, input_mask],outputs=output)  # FC: based on the code in the lines above, a model is now constructed and passed into the variable model
    model.compile(tf.keras.optimizers.Adam(learning_rate=1e5),loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    return model

with strategy.scope():
    model = build_model()
    model.summary()

# print("model.layers[2]:-------". model.layers[2])
# model.layers[2].trainable=True

for key in train_input.keys():
    train_input[key] = train_input[key][:,:max_len]

print("train the model now")
early_stop = tf.keras.callbacks.EarlyStopping(patience=3,restore_best_weights=True)

model.fit(train_input, train.label.values, epochs = 3, verbose = 1, validation_split = 0.01,
         batch_size=16*strategy.num_replicas_in_sync,
          callbacks=[early_stop])
print("Training is completeted")

model.save("saved_model/trackers/1")```

Context: Try using model.save_weights() instead of model.save()


Compute gradients across two layers using gradients calculated from a previous layer using tf.gradients or tf.GradientTape

Question:
I want to use the gradients of one layer to calculate the gradients of the layer that comes before it.
My motivation for doing this is, when I tried to use model parallelism using tf.device, I found out that backpropagation has been running on CPU. The entire Backprop started running on a chosen tf.device only after I wrapped the call to GradientTape(when it computes the gradient) within the tf.device context manager. Since the model is split, I want the backprop of each partition to execute on the device where that partition is placed.
Ideally, I would like to find out a method with which this oversimplified pseudocode is possible.
with tf.device(device_3):
   grad_3 = tf.gradients(loss, trainable_vars_of_partition_3)

with tf.device(device_2):
   grad_2 = tf.gradients(grad_3, trainable_vars_of_partition_2)

with tf.device(device_1):
   grad_1 = tf.gradients(grad_2, trainable_vars_of_partition_1)

grads = concat(grad_1, grad_2, grad_3)

If something like this exists then I would be overjoyed if you could point me in the right direction.
Unfortunately, I could not find something as simple as this. The next best approach that I could think of was using the gradients of one layer to find the gradients of a layer that comes before it. Using chain rule and backpropagation, I feel that this should be possible.
I created this toy example, solving which is the first step towards the final goal.
Let’s say we have a model with 3 dense layers without activations functions. X, Y as defined as follows:
x = tf.concat([tf.random.uniform([1, 10], minval=0, maxval=0.25),
               tf.random.uniform([1, 10], minval=0.25, maxval=0.5),
               tf.random.uniform([1, 10], minval=0.5, maxval=0.75),
               tf.random.uniform([1, 10], minval=0.75, maxval=1.),
                ], axis = 0)

y = tf.constant(0., shape=[4, 1])

d1 = tf.keras.layers.Dense(5, name='d1')
d2 = tf.keras.layers.Dense(2, name='d2')
d3 = tf.keras.layers.Dense(1, name='d3')

I am using a tf.function in this toy example but an answer with eager mode enabled, using GradientTape will also be appreciated.
@tf.function
def tf_func(x, y, d1, d2, d3):
    # Using shortforms of these function helped the code look neater and more readable to me.
    g = tf.gradients
    rs = tf.reduce_sum
    rm = tf.reduce_mean

    o1 = d1(x)
    o2 = d2(o1)
    o3 = d3(o2)

    l = tf.reduce_mean(tf.square(o3 - y))

    w3, w2, w1 = d3.trainable_variables, d2.trainable_variables, d1.trainable_variables

    tf.print('actual grads' + '=' * 80)

    dl_dw3 = g(l, w3)

    dl_dw2 = g(l, w2)
    tf.print('dl_dw2: \n', dl_dw2)

    dl_dw1 = g(l, w1)

    tf.print()
    tf.print()

    tf.print('reference grads' + '=' * 80)
    dl_do1 = g(l, o1)
    dl_do2 = g(l, o2)
    tf.print('dl_do2: \n', dl_do2)
    dl_do3 = g(l, o3)

    dl_dw1 = g(l, w1)
    dl_dw2 = g(l, w2)
    dl_dw3 = g(l, w3)

    do3_o2 = g(o3, o2)
    do2_do1 = g(o2, o1)

    do3_w3 = g(o3, w3)
    do2_w2 = g(o2, w2)
    do1_w1 = g(o1, w1)

    tf.print('testing chain_rule method' + '=' * 80)

    # Added a 't' before derivatives to differentiate between ref_grads and grads obtained using chain rule

    tdl_do3 = g(l, o3) # same as ref_grads

    tdo3_dw3 = g(o3, w3) # same as ref_grads
    tdl_dw3 = [rm(tdl_do3) * tdo3_dw3[0], rm(tdl_do3) * tdo3_dw3[1]] # same as actual grads

    tdo3_do2 = g(o3, o2) # same as ref_grads

    tdl_do2 = tdo3_do2 * rm(tdl_do3, axis=0)  # same as ref_grads
    tf.print('tdl_do2: \n', tdl_do2)

    tdo2_dw2 = g(o2, w2)
    tf.print('tdo2_dw2: \n', tdo2_dw2)

    tdl_dw2 = [tdo2_dw2[0] * rm(tdl_do2, axis=[1]), tdo2_dw2[1] * rm(tdl_do2, axis=[1])]
    tf.print('tdl_dw2: \n', tdl_dw2)

    return None

tf_func(x, y, d1, d2, d3)

The output was:
actual grads================================================================================
dl_dw2:
 [[[-3.04819393 -1.30051827]
 [5.02123785 2.14232159]
 [-0.260933906 -0.111328]
 [5.87596226 2.50699162]
 [1.9655633 0.838611722]], [-4.69162369 -2.0016911]]

reference grads================================================================================
dl_do2:
 [[[-0.43842113 -0.187053293]
 [-0.889310718 -0.379426271]
 [-1.41650343 -0.604354143]
 [-1.94738865 -0.830857456]]]

testing chain_rule method================================================================================
tdl_do2:
 [[[-0.43842113 -0.187053293]
  [-0.889310718 -0.379426271]
  [-1.41650343 -0.604354143]
  [-1.94738865 -0.830857456]]]
tdo2_dw2:
 [[[2.10966444 2.10966444]
 [-3.48670244 -3.48670244]
 [0.22972326 0.22972326]
 [-3.95618558 -3.95618558]
 [-1.3790133 -1.3790133]], [4 4]]
tdl_dw2:
 [[[-2.47443795 -1.05572414]
 [4.08957386 1.74482536]
 [-0.26944378 -0.114958748]
 [4.64023352 1.97976542]
 [1.61745286 0.690089643]], [[-4.69162369 -2.0016911]]]

For some reason, gradients wrt weights in tdl_dw2 and dl_dw2 differ slightly. Every value in tdl_dw2 is slightly less than dl_dw2 even though the gradients wrt biases are the same. I cannot figure out why.
The gradient of loss wrt to w3 is as expected.
I used tf.reduce_mean to replicate what tf.gradients was doing internally as far as I understand. Please correct me if I am wrong.
From Tensorflow’s documentations:

gradients() adds ops to the graph to output the derivatives of ys with respect to xs. It returns a list of Tensor of length len(xs) where each tensor is the sum(dy/dx) for y in ys and for x in xs.

tf.gradients constructs symbolic derivatives of sum of ys w.r.t. x in xs.

Any guidance or help will be greatly appreciated, thank you.
Some Similar StackOverflow questions(there are many more):

python - Compute gradients across two models - Stack Overflow
python - Is it possible to acquire an intermediate gradient? (Tensorflow) - Stack Overflow
automatic differentiation - Breaking TensorFlow gradient calculation into two (or more) parts - Stack Overflow

Here is a colab notebook with the code:

      colab.research.google.com

Google Colaboratory

Context:
There’s a lot going on in this question.
But you shouldn’t need to go that low level to do this.
Given Alex’s response 1 I hope we can get that issue fixed eventually.
3 things that may help here in the mean time:

GradientTape.gradient accepts an output_gradients argument. To run your gradient in stages you can do something like this:

with tf.GradientTape() as tape1:
   x = chunk1(input)

with tf.GradientTape() as tape2:
  tape2.watch(x)
  y = chunk2(x)
  loss = loss_fn(y, y_true)

g2, gx = tape2.gradient(loss, [chunk2.variables, x])

g2 = tape1.gradient(x, [chunk1.variables], output_gradient=gx)

I should add an example of this to the Advanced automatic differentiation guide 1
I’m going to guess that if you wrap your with tf.device over the with tf.GradientTape then the gradient operations go on that device.

Another thing that could work would be to use @tf.custom_gradient. I’m pretty sure you can use a with tf.device(), tf.GradientTape() as tape: in the custom gradient function.

For model parallelism mesh approaches have been gaining popularity. Where instead of assigning layers to devices you split layers across devices. But I don’t have an off-the-shelf solution for that.


Save a tensorflow model with a transformer layer

Question:
Hi
I trained a model with the following architecture:
bert_config = BertConfig.from_pretrained(MODEL_NAME)
bert_config.output_hidden_states = True
backbone = TFAutoModelForSequenceClassification.from_pretrained(MODEL_NAME,config=bert_config)

input_ids = tf.keras.layers.Input(shape=(MAX_LENGTH,), name='input_ids', dtype='int32')
features = backbone(input_ids)[1][-1]
pooling =  tf.keras.layers.GlobalAveragePooling1D()(features)
dense = tf.keras.layers.Dense(len(label2id), name='output',activation=tf.nn.softmax)(pooling)

model = tf.keras.Model(inputs=[input_ids], outputs = [dense])

and saved model in different ways. The first one is
model.save_weights('/content/drive/MyDrive/weights/weights')

and the second one is
model.save_weights('/content/drive/MyDrive/weights/weights.h5')

So, I am able to load my model (model.load_weights()) from both of these options without any error. Moreover, inference is fine. In short, everything works as I expect.
But if I start a new session and load my model again then inference is bad, like model has random weights instead of my saved weights.
I was trying other options of saving models as well, but they do not work also. Probably there is a special way to save a model with transformer layer?
Thanks in advance!

Context: Do you have experienced the same problem with model.save?


Build 32-bit tensorflowlite library on Windows

Question: According to the official suggestion, I can build a 64-bit library “.dll” and "lib"  on Windows platform, but how to compile a 32-bit library?

Context:
Please check

    How to build 32bit tensorflow on Windows? General Discussion

    hi,is anyone tell me how to build 32bit tensorflow on Windows?version 1.12.0 is just fine.


Differences in results between saving in .h5 format and weights only

Question:
I create a model based on efficientNet_b0, when I tried to save it in the default keras format it gave the following error:
KeyError: “Failed to add concrete function ‘b’__inference_EfficientNet_layer_call_fn_1379410’’ to object-based SavedModel as it captures tensor <tf.Tensor: shape=(), dtype=resource, value=> which is unsupported or not reachable One reason could be that a stateful object or a variable that the function depends on is not assigned to an attribute of the serialized trackable object (see SaveTest.test_captures_unreachable_variable).”
So, I saved it to .h5, but the result was not the same as the one obtained before (this on the same dataset).
Previously obtained result:

Confusion Matrix
231 0
0 97

Result obtained with the load in .h5 format:

Confusion Matrix
231 62
0 35

So, I try this problem, I just saved the weights, and I recreated the architecture, and the result is back to normal.
Any tips where I might be going wrong? And how to fix the error of saving by keras default mode?

Context:
@Staub,
Can you take a look at this github issue 2 which discusses about similar KeyError. Also can you take a look at this guide 2. Thanks!


None of the MLIR Optimization passes are enabled

Question:
I am training an object detection model with  Tensorflow Object Detection API following this tutorial 62, But during training the model, it always gets stuck at these warning and it never updates, even after hours, how to avoid this!?
 2021-06-22 05:03:36.742721: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    2021-06-22 05:03:36.752083: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2299995000 Hz

Context:
Are you in the same case as:

      github.com/tensorflow/models

      Object detection API stuck on None of the MLIR optimization passes are enabled (registered 2) 971

        opened

        Jan 14, 2021

          closed

        Jan 28, 2021

          ysefwakil

        models:research
        stat:awaiting response
        type:bug

    # Prerequisites

Please answer the following questions for yourself before sub…mitting an issue.

- [X ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
- [X ] I am reporting the issue to the correct repository. (Model Garden official or research directory)
- [X ] I checked to make sure that this issue has not been filed already.

## 1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/official/...

## 2. Describe the bug

Not allowing me to start training

## 3. Steps to reproduce

Followed the tutorial posted https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/

## 4. Expected behavior

I expected to start training but it is stuck on that line and not continuing further

## 5. Additional context

None

## 6. System information

- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows x64
- TensorFlow installed from (source or binary): Pip Install
- TensorFlow version (use command below): 2.4.0
- Python version: 3.8
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version: Cuda 11.0 Cudnn 8.0
- GPU model and memory: GTX1070 8GB

I am following the steps and guide for the Tensorflow object detection API tutorial (https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/) and everything worked up till now where It wont commence training. Any Ideas?

![image](https://user-images.githubusercontent.com/48973627/104555196-5246e280-56a2-11eb-89a0-e7d564daf736.png)


Question:
Can we have a dedicated Discussion tab in keras 1 repositories?

Examples: Discussions · facebookresearch/detectron2 · GitHub 1

Examples: Discussions · ultralytics/yolov5 · GitHub 1

Context:
I think that we need to be able to define a not overlapping scope to evaluate a discussion Tab in ecosystem repos as we already have some overlap between GitHub issues, this forum and stackoverflow.
/cc @Joana @thea


I have a problem about object detector model training

Question:
This is the link to my google Colab  project
: Google Colab 1
the problem started by when I wanted to trained the object detector model but It was stuck with this problem :

image1849×435 35.6 KB

I don’t know how to solve it out, anybody please help me
Thanks

Context:
@khanhlvg Would you mind If you could help me about the code
because I have learn from your video : Train a custom object detection model using your data - YouTube 1


Question: Are there any plans to support Python 3.10?

Context: Python 3.10 will be released in October 2021, so it will be supported in the next TF release after that, assuming all dependencies will be migrated by release time.


TFLite Classification Model Grayscale Input in Java

Question:
Hello,
I trained a Keras CNN model for a classification task using grayscale images (48x48x1). After training and saving the model, I now would like to use the model in an Android application, so I converted the trained model to a TFLite model.
After conversion, I did two things in order to confirm that the converted model works properly:

I tested running inferences on some grayscale test images using the TFLite model in Python, and got proper and expected results (using the same dimensions for the input as I used for training).

I inspected the TFLite model in Netron and confirmed that it expects an input of the following format: float32[1, 48, 48, 1].

The issue I am facing now is with loading input images to the TFLite model using Java in Android Studio. Since my model expects grayscale input images (1 channel), the total input bytes for one frame are supposed to be 48x48x1x4= 9216 bytes (the last 4 multiplied is because of the input being of FLOAT32 format). The problem is that the TensorImage input format, which is what the TFLite model supports, only supports RGB input images, which isn’t compatible with single-channel input images. In addition, Bitmap doesn’t even support storing grayscale images, so I wouldn’t know how to store any grayscale input images into a Bitmap and not have an extra number of bytes that would later cause issues.
I am not an expert in Android development and have slowly been learning and building things, so I understand there might be easy solutions that I’m overlooking. I would appreciate any help that comes my way!
Thanks,
Ahmad

Context:
Hi @ahmadchalhoub99 ,
Thanks for the detailed explanation. This issue has been taken care of about a year ago with the introduction of TensorFlow Lite Support TransformToGrayScaleOp.java 1 operator. Check inside the file to see how this is handled.
So inside your code you can use the TensorFlow Lite Support library 1 using the Basic image manipulation and conversion 1 adding the operator to the ImageProcessor.
I hope it helps.
Tag me if you need something more.
Regards


Giving error while processing float values

Question:
def gaussian_exp(inp_val):
        """
        Expand values to gaussian basis

        """
        guass_filter = tf.range(-5, 5 + 1, 1)
        return tf.math.exp(-(tf.expand_dims(inp_val, axis=-1) - guass_filter)**2 /(1**2))

The above code works fine for integer values, but gives an error while processing float values, any idea?

Context:
It Is explained in the doc as a constrain:

      TensorFlow

tf.math.exp  |  TensorFlow Core v2.7.0 1

  Computes exponential of x element-wise. \\(y = e^x\\).


Can TFJS make use of the TPU on a Pixel 6?

Question: I’m wondering if, as phones like Pixel 6 become common, whether TFJS in a browser will run much faster than TFJS currently does on modern phones. If the answer is yes are there any benchmarks? If not, is it because the TPU isn’t compatible with a GPU? Or is it?

Context:
I don’t think that currently we have a W3C standard to access to TPU.
Also the WIP webnn, in the current status, It is still limited to CPU/GPU devices:
https://www.w3.org/TR/webnn/#programming-model-device-selection 1
For Coral TPU that is a quite different use case see:

    TFJS and Coral TPU General Discussion

    This was the most updated status:


Question:
Hi, I’m new to TensorFlow. While applying tutorial code from ’ Using tf.data for finer control’ section of the ‘Load and Preprocess data’ tutorial to my own data that have two categories, I obtained error messages apparently related to loss functions. The two loss functions below compile but produce errors during model fitting:
SparseCategoricalCrossentropy results in ‘InvalidArgumentError:  Received a label value of 2 which is outside the valid range of [0, 2).’ This is the loss function used in the tutorial.
BinaryCrossentropy results in ‘ValueError: logits and labels must have the same shape ((None, 2) vs (None, 1))’. I tried this as an alternative.
I should also note that I was able to successfully fit a model to my data using the methods described in the first part of the tutorial. It’s only the approach used in the second part of the tutorial that raises errors.
Thanks in advance for any advice.

Context: What are the values of your labels?


Question:
Hello!
I’m pretty new to TensorFlow and I am trying to classify german words to their gramatical gender.
My problem is that TensorFlow crashes with an error that I can’t find online.

My python code
import tensorflow as tf
import numpy as np
from tensorflow.keras.layers import TextVectorization
from tensorflow.keras import layers

batch_size = 1

raw_train_ds = tf.keras.preprocessing.text_dataset_from_directory(
    "D:/artikelguesser/train",
    labels='inferred',
    class_names=["feminine","masculine","neuter"],
    batch_size=batch_size,
    validation_split=0.2,
    subset="training",
    seed=25565
)
raw_val_ds = tf.keras.preprocessing.text_dataset_from_directory(
    "D:/artikelguesser/train",
    labels='inferred',
    class_names=["feminine","masculine","neuter"],
    batch_size=batch_size,
    validation_split=0.2,
    subset="validation",
    seed=25565
)

raw_test_ds = tf.keras.preprocessing.text_dataset_from_directory(
    "D:/artikelguesser/test", batch_size=batch_size
)

print(f"Number of batches in raw_train_ds: {raw_train_ds.cardinality()}")
print(f"Number of batches in raw_val_ds: {raw_val_ds.cardinality()}")
print(f"Number of batches in raw_test_ds: {raw_test_ds.cardinality()}")

max_features = 32906
embedding_dim = 128
sequence_length = 20

vectorize_layer = TextVectorization(
    standardize=None,
    max_tokens=max_features,
    output_mode="int",
    split=None,
    output_sequence_length=sequence_length,
)

text_ds = raw_train_ds.map(lambda x, y: x)
vectorize_layer.adapt(text_ds)

def vectorize_text(text, label):
    text = tf.expand_dims(text, -1)
    return vectorize_layer(text), label

train_ds = raw_train_ds.map(vectorize_text)
val_ds = raw_val_ds.map(vectorize_text)
test_ds = raw_test_ds.map(vectorize_text)
print("Vectorized!")

train_ds = train_ds.cache().prefetch(buffer_size=10)
val_ds = val_ds.cache().prefetch(buffer_size=10)
test_ds = test_ds.cache().prefetch(buffer_size=10)
print("Prefetched!")

inputs = tf.keras.Input(shape=(None,), dtype="int64")
x = layers.Embedding(max_features, embedding_dim)(inputs)
x = layers.Dropout(0.5)(x)

x = layers.Conv1D(128, 7, padding="valid", activation="relu", strides=3)(x)
x = layers.Conv1D(128, 7, padding="valid", activation="relu", strides=3)(x)
x = layers.GlobalMaxPooling1D()(x)

x = layers.Dense(128, activation="relu")(x)
x = layers.Dropout(0.5)(x)

predictions = layers.Dense(3, activation="sigmoid", name="predictions")(x)

model = tf.keras.Model(inputs, predictions)

model.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
print("Model compiled!")

model.fit(train_ds, validation_data=val_ds, epochs=3)

print("DONE!")

Output
Found 41132 files belonging to 3 classes.
Using 32906 files for training.
2022-01-23 21:30:44.635916: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-01-23 21:30:45.176399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4639 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 980 Ti, pci bus id: 0000:03:00.0, compute capability: 5.2
Found 41132 files belonging to 3 classes.
Using 8226 files for validation.
Found 4569 files belonging to 3 classes.
Number of batches in raw_train_ds: 32906
Number of batches in raw_val_ds: 8226
Number of batches in raw_test_ds: 4569
Vectorized!
Prefetched!
Model compiled!
Epoch 1/3
2022-01-23 21:31:29.712176: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8101
2022-01-23 21:31:30.866313: F tensorflow/stream_executor/cuda/cuda_dnn.cc:570] Check failed: cudnnSetTensorNdDescriptor(handle_.get(), elem_type, nd, dims.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)batch_descriptor: {count: 1 feature_map_count: 128 spatial: 1 0  value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX}

Version stuff

GPU: GTX 980 TI
GPU driver: 511.23
TF: 2.7.0
OS: Windows 10.0.19043
CUDA: 11.2
cuDNN: 8.1.0.77
Python: 3.9.6

I’ve noticed that there are two folders in C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\lib: Win32 and x64. I’m running a 64 bit Windows. Is this normal?

Context: What Is you TF version?


Question:
So, I have been exploring NLP of recent.
There’s this one project that I have been planning to do for a while, but each time I put it on the sidelines.
It’s translating English to Chinese and Chinese to English.
The biggest challenge am encountering is preprocessing Chinese characters. I have been looking around, but unable to find any information.
How do I preprocess and tokenize Chinese and similar languages characters?

Context:
We had a thread at:

      github.com/keras-team/keras

      how to use TextVectorization to vectorize Chinese text  1

        opened

        Sep 25, 2020

          baiziyuandyufei

          type:support

    In Chinese text, there is no whitespace between words, so when I use TextVectori…zation.adapt(train_dataset), I can only get Sentence-level vocabulary.  the code I used is [https://keras.io/examples/nlp/text_classification_from_scratch/](https://keras.io/examples/nlp/text_classification_from_scratch/)


Occasionally skip weight updates inside train_step of Tensorflow Keras

Question:
I’m building an RNN Model where I would like to do weight updates only when some condition, based on the state of the RNN, is met:
class CustomModel(keras.Model):
    def train_step(self, data):
        x, y = data
        # Perform forward pass and compute gradients

        if (some condition is met):
            #Apply gradients

I tried an if condition that only updates based on the value of a 1 x 1 tf.Variable t_step like  if t_step > 5.  but got the error below:

OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

t_step is essentially the time step of the input, but when eager execution is disable it seems like I cannot use it in the if condition.

Context: Have you tried with tf.cond  |  TensorFlow Core v2.7.0 1 ?


Update tensorflow-federated to match tensorflow-macos==2.7.0 and tensorflow-metal==0.3.0

Question:
I was able to use tensorflow-federated==0.19.0 with tensorflow-macos==2.5.0 and tensorflow-metal==0.2.0 on an Intel-based MacBook Pro with GPU.
Now tensorflow-macos==2.7.0 and tensorflow-metal==0.3.0 has been upgrade to tf 2.7 and tensorflow-federated==0.19.0 still depends on tf 2.5.
Is there any roadmap to update tensorflow-federated to tf 2.7 ?

Context:
I can force the installation with:
SYSTEM_VERSION_COMPAT=0 pip install tensorflow-macos==2.7.0 tensorflow-metal==0.3.0
SYSTEM_VERSION_COMPAT=0 pip install tensorflow-federated==0.19.0

and got an error:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow-macos 2.7.0 requires tensorflow-estimator<2.8,~=2.7.0rc0, but you have tensorflow-estimator 2.5.0 which is incompatible.

It appears to me that my tff examples will still work. But it would be really nice to have an update on tensorflow-federated with dependency on tensorflow-estimator>=2.7.0


Question:
Hi all,
I am trying to set up a GB classifier on TFX for the first time. I am able to run smoothly with the interactive context. However when trying to run it on Kubeflow on GCP, I am having an issue with the transform component. Here are a few questions, which could hopefully solve the problems I am encountering:

What is the recommended environment and/or VM to set up TFDF with TFX?
When creating the pipeline on Kubeflow (with CLI: tfx pipeline create […]), wheels are being created for the transform and trainer component in a /tmp folder in (such as tfx_user_code_Transform-[…]). What purpose are those wheels serving and how to indicate where to store/retrieve them? I dont exactly know why and my set up might be wrong but the Transform component, when ran on Kubeflow, is looking for those wheels in the wrong location ( ings://[…]/[…]/_wheels/[…]). More context is given here 1

Is there any code snippet or architecture code that we could follow to set up a TFDF model on TFX. Or a estimator.GradientBoostedClassifier that I can’t seem to be able to set up properly.

Any help would be greatly appreciated
Armand

Context:
We have an example of TFDF here:

      github.com

    tensorflow/tfx/blob/master/tfx/examples/penguin/penguin_pipeline_local.py 3

    # Copyright 2020 Google LLC. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Penguin example using TFX."""

import multiprocessing
import os
import socket
import sys
from typing import List, Optional

  This file has been truncated. show original


VAE with Tensorflow probability library. Problems understanding example

Question:
Hello,
I am trying to replicate the approach to create a VAE as described here 2.
However, I don’t understand the following piece of code and how to translate it from images (as per example) to the data I have, which is a big csv file with lots of columns and rows.
def _preprocess(sample):
  image = tf.cast(sample['image'], tf.float32) / 255.  # Scale to unit interval.
  image = image < tf.random.uniform(tf.shape(image))   # Randomly binarize.
  return image, image

train_dataset = (datasets['train']
                 .map(_preprocess)
                 .batch(256)
                 .prefetch(tf.data.AUTOTUNE)
                 .shuffle(int(10e3)))
eval_dataset = (datasets['test']
                .map(_preprocess)
                .batch(256)
                .prefetch(tf.data.AUTOTUNE))

I don’t understand what the _preprocess function is exactly doing and why this is required for the model to work. This prevents me to adapt this solution for my csv dataset.
I hope this is the correct category to ask for help.

Context:
Hi @Exitare and welcome

 Exitare:

def _preprocess(sample):

This looks like an image preprocessing helper function.

 Exitare:

  image = tf.cast(sample['image'], tf.float32) / 255.  # Scale to unit interval.

This casts the tensor to dtype tf.float32 and normalizes the floating point values to the [0, 255] range.

 Exitare:

  image = image < tf.random.uniform(tf.shape(image))   # Randomly binarize.

Note: “The pixel intensities of MNIST images are almost binary.” (e.g. 255 is white, 0 is black).

 Exitare:

how to translate it from images (as per example) to the data I have, which is a big csv file with lots of columns and rows.

Check out the 3rd example 2 from the Intro to Autoencoders tutorial - it uses electrocadriogram data from a CSV file. Let us know if it helps.


Parallelising custom function in tensorflow using graph execution

Question:
Hi
I’m trying to do sensitivity analysis (forward mode autodiff) on a matrix and was hoping to parallelise the computations using Tensorflow. Here’s the code that I’m using to test if something like this is possible in TF:
def _forward(X, dX, W1, W2):
    Z1 = tf.matmul(X, tf.transpose(W1))
    dZ1 = tf.matmul(dX, tf.transpose(W1))

    A1 = tf.tanh(Z1)
    dA1 = tf.multiply(tf.expand_dims(1-tf.square(tf.tanh(Z1)), axis=1), dZ1)

    Z2 = tf.matmul(A1, tf.transpose(W2))
    dZ2 = tf.matmul(dA1, tf.transpose(W2))

    return Z2, tf.squeeze(dZ2, axis=-1)

In the code above, the evaluation of Z1 and dZ1 is independent of each other (same thing for A1, dA1, etc. etc.) so I was hoping to run these pair of statements in parallel. I wrapped this function around tf.function and was hoping for speedup as compared to standard way of computing gradients (forward and backprop) because now I’ll be running half the calculations in parallel. However, I don’t see any speedups and both codes take the same time to execute.
I don’t know if it’s possible to do what I’m trying here. Any help would be appreciated.
Thanks

Context:
Have you tried to jit compile It:

      TensorFlow

XLA: Optimizing Compiler for Machine Learning  |  TensorFlow 1

  XLA is a compiler-based linear algebra execution engine.


TensorBoard unable to see Debugger V2 data

Question:
Hi,
I’m using tf-nightly-gpu and tb-nightly and am trying to debug a model that is numerically unstable. I’d like to see where NaNs are occurring in the compute graph. So I’ve enabled debugging:
tf.debugging.experimental.enable_dump_debug_info("debug", tensor_debug_mode="FULL_HEALTH", circular_buffer_size=-1)

I run an epoch of training and then try to run tensorboard:
tensorboard --logdir=debug

I enable the Debugger V2 plugin but am always greeted with:
Debugger V2 is inactive because no data is available.

Here are the files written to the debug/ directory:
root@ce9fb22e47b0:/projects/FasterRCNN/tf2# ls debug -alh
total 195M
drwxr-xr-x 2 root root 4.0K Jan  4 18:37 .
drwxrwxr-x 7 1000 1000 4.0K Jan  4 18:40 ..
-rw-r--r-- 1 root root 5.6M Jan  4 18:37 tfdbg_events.1641320809.ce9fb22e47b0.execution
-rw-r--r-- 1 root root 109M Jan  4 18:37 tfdbg_events.1641320809.ce9fb22e47b0.graph_execution_traces
-rw-r--r-- 1 root root  18M Jan  4 18:37 tfdbg_events.1641320809.ce9fb22e47b0.graphs
-rw-r--r-- 1 root root   71 Jan  4 18:26 tfdbg_events.1641320809.ce9fb22e47b0.metadata
-rw-r--r-- 1 root root 7.1M Jan  4 18:37 tfdbg_events.1641320809.ce9fb22e47b0.source_files
-rw-r--r-- 1 root root  84K Jan  4 18:37 tfdbg_events.1641320809.ce9fb22e47b0.stack_frames
-rw-r--r-- 1 root root 3.1M Jan  4 18:40 tfdbg_events.1641321476.ce9fb22e47b0.execution
-rw-r--r-- 1 root root  18M Jan  4 18:40 tfdbg_events.1641321476.ce9fb22e47b0.graph_execution_traces
-rw-r--r-- 1 root root  29M Jan  4 18:40 tfdbg_events.1641321476.ce9fb22e47b0.graphs
-rw-r--r-- 1 root root   71 Jan  4 18:37 tfdbg_events.1641321476.ce9fb22e47b0.metadata
-rw-r--r-- 1 root root 7.1M Jan  4 18:40 tfdbg_events.1641321476.ce9fb22e47b0.source_files
-rw-r--r-- 1 root root  85K Jan  4 18:40 tfdbg_events.1641321476.ce9fb22e47b0.stack_frames

There is no indication of any other error from either TensorBoard or TensorFlow during training.
Thank you,
Bart

Context: I got the same error and cannot find any useful docs in the internet. Hope anyone can help on it!


Question:
Hi,
Getting the following error
TypeError: ‘_VariantDataset’ object is not subscriptable
when trying to access the first element like img[‘image’] from the dataset. How can I access first element of the dataset?
Thanks

Context:
Have  you tried with:

      TensorFlow

tf.data.experimental.get_single_element  |  TensorFlow Core v2.5.0 4

  Returns the single element in dataset as a nested structure of tensors.

Or you can use an iterator:

      TensorFlow

tf.data.Iterator  |  TensorFlow Core v2.5.0 2

  Represents an iterator of a tf.data.Dataset.


How to augment training data ‘on the fly’ with multiclass segmentation using Keras

Question:
I am trying to follow through this code sample: Multiclass semantic segmentation using DeepLabV3+
The author is not augmenting the data in any way. The dataset used here (CIHP) has training, validation, and test images and both training and validation have corresponding masks for the objects of interest.
How can we run augmentations on the data during training? Can you guys provide some examples?

Context:
Here is some nice guide. You should modify your data_generator method.

      TensorFlow

Data augmentation  |  TensorFlow Core 1