Lambda Layers in tf.keras. Lambda layers are useful when you need…
By: Rahul Bhadani
Tags: Tensorflow2, Data Science, Lambda, Machine Learning, TensorFlow

Introduction

Lambda layers are useful when you need to do some operations on the previous layer but do not want to add any trainable weight to it. Lambda layer is an easy way to customize a layer to do simple arithmetic. Let say you want to add your own activation function (which is not built-in Keras) to a layer. Then you first need to define a function that will take the output from the previous layer as input and apply a custom activation function to it. We then pass this function to the lambda layer.

From Tensorflow documentation

The Lambda layer exists so that arbitrary TensorFlow functions can be used when constructing Sequential and Functional API models. Lambda layers are best suited for simple operations or quick experimentation. For more advanced use cases, subclassing keras.layers.Layer is preferred. One reason for this is that when saving a Model, Lambda layers are saved by serializing the Python bytecode, whereas subclassed Layers are saved via overriding their get_config method and are thus more portable. Models that rely on subclassed Layers are also often easier to visualize and reason about.

Example 1

# add a x -> x^2 layer

model.add(Lambda(lambda x: x ** 2))

Example 2a

# add a layer that returns the concatenation

# of the positive part of the input and

# the opposite of the negative part def antirectifier(x):

x -= K.mean(x, axis=1, keepdims=True)

x = K.l2_normalize(x, axis=1)

pos = K.relu(x)

neg = K.relu(-x)

return K.concatenate([pos, neg], axis=1) model.add(Lambda(antirectifier))

Example 2b

def linear_transform(x):

v1 = tf.Variable(1., name='multiplier')

v2 = tf.Variable(0., name='bias')

return x*v1 + v2 linear_layer = Lambda(linear_transform)

model.add(linear_layer)

model.add(keras.layers.Dense(10, activation='relu'))

model.add(linear_layer) # Reuses existing Variables

Example 3

from keras.layers import Lambda

from keras import backend as K

# defining a custom non linear function

def activation_relu(inputs):

return K.maximum(0.,inputs) # call function using lambda layer

squashed_output = Lambda(activation_relu)(inputs) # where inputs are output from previous layer

Example 4

def output_of_lambda(input_shape):

return (input_shape[0], 1, input_shape[2]) def mean(x):

return K.mean(x, axis=1, keepdims=True) model.add(Lambda(mean, output_shape=output_of_lambda))

Example 5

Let say I want to build a custom layer that computes the element-wise euclidean distance between two input tensors, I would define the function to compute the value itself, as well as one that returns the output shape from this function.

def euclidean_distance(vecs):

x, y = vecs

return K.sqrt(K.sum(K.square(x - y), axis=1, keepdims=True)) def euclidean_distance_output_shape(shapes):

shape1, shape2 = shapes

return (shape1[0], 1)

Then call these functions using the lambda layer shown as follows:

lhs_input = Input(shape=(VECTOR_SIZE,)) lhs = dense(1024, kernel_initializer="glorot_uniform", activation="relu")(lhs_input) rhs_input = Input(shape=(VECTOR_SIZE,)) rhs = dense(1024, kernel_initializer="glorot_uniform", activation="relu")(rhs_input) sim = lambda(euclidean_distance, output_shape=euclidean_distance_output_shape)([lhs, rhs])

References