TensorFlow Lite on Android
Tags: TensorFlow, Tensorflow Lite, Object Detection, Object Recognition, Android

TensorFlow Lite?

TensorFlow Lite is a set of tools to help developers run TensorFlow models on mobile, embedded, and IoT devices. It enables on-device machine learning inference with low latency and a small binary size.

It’s currently supported on Android and iOS via a C++ API, as well as having a Java Wrapper for Android Developers. Additionally, on Android Devices that support it, the interpreter can also use the Android Neural Networks API for hardware acceleration, otherwise it will default to the CPU for execution. In this article I’ll focus on how you use it in an Android app.

TensorFlow Lite consists of two main components:

The TensorFlow Lite interpreter, which runs specially optimized models on many different hardware types, including mobile phones, embedded Linux devices, and microcontrollers.

The TensorFlow Lite converter, which converts TensorFlow models into an efficient form for use by the interpreter, and can introduce optimizations to improve binary size and performance.

These are the details about the abbreviation

Latency: there’s no round-trip to a server

Privacy: no data needs to leave the device

Connectivity: an Internet connection isn’t required

Power consumption: network connections are power hungry

In the below I am going to share how to implement and use of TensorFlow’ slibrary lite in Android Project.

NOTE → Before going below codes you can check github link https://github.com/amitshekhariitbhu/Android-TensorFlow-Lite-Example for implementation of TensorFlow’s lite I also follow his link. It’s a best and maintain project which is available in github.

Step 1. Add dependency

dependencies {

implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly'

}

Step 2. (OPTIONAL)for achieving the x86 , x86_64 , and arm32 ABIs , add the below dependency in your gradle.

android {

defaultConfig {

ndk {

abiFilters 'armeabi-v7a', 'arm64-v8a'

}

}

}

Step 3 -> Create layout for Camera view activity_ tensorflow.xml

<?xml version="1.0" encoding="utf-8"?>

<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"

xmlns:tools="http://schemas.android.com/tools"

android:id="@+id/activity_main"

android:layout_width="match_parent"

android:layout_height="match_parent"

android:paddingBottom="@dimen/activity_vertical_margin"

android:paddingLeft="@dimen/activity_horizontal_margin"

android:paddingRight="@dimen/activity_horizontal_margin"

android:paddingTop="@dimen/activity_vertical_margin"

tools:context="com.example.myapp.tancerflow.TenserFlowActivity">



<com.wonderkiln.camerakit.CameraView

android:id="@+id/cameraView"

android:layout_width="300dp"

android:layout_height="300dp"

android:layout_gravity="center|top" />





<LinearLayout

android:layout_width="match_parent"

android:layout_height="80dp"

android:layout_gravity="center|top"

android:layout_marginTop="300dp"

android:gravity="center"

android:orientation="horizontal">



<ImageView

android:id="@+id/imageViewResult"

android:layout_width="75dp"

android:layout_height="75dp"

android:padding="2dp" />



<TextView

android:id="@+id/textViewResult"

android:layout_width="match_parent"

android:layout_height="80dp"

android:fadeScrollbars="false"

android:gravity="center"

android:maxLines="15"

android:scrollbars="vertical"

android:textColor="@android:color/black" />



</LinearLayout>





<Button

android:id="@+id/btnToggleCamera"

android:layout_width="match_parent"

android:layout_height="48dp"

android:layout_gravity="bottom|center"

android:layout_marginBottom="50dp"

android:text="toggle_camera"

android:textAllCaps="false"

android:textColor="@android:color/black" />



<Button

android:id="@+id/btnDetectObject"

android:layout_width="match_parent"

android:layout_height="48dp"

android:layout_gravity="bottom|center"

android:text="detect_object"

android:textAllCaps="false"

android:textColor="@android:color/black"

android:visibility="gone" />



</FrameLayout>

Note → For dependency Camera view implement gradle in your project

implementation 'com.wonderkiln:camerakit:0.13.1'

Step 4 → Add mobilenet_quant_v1_224.tflite file in your assets folder.

for downloading file check in github https://github.com/amitshekhariitbhu/Android-TensorFlow-Lite-Example/tree/master/app/src/main/assets

Step 5 → Create a TensorFlowActivity.class and give the correct path of tflite file in the place of modelPath.

below the whole class should be like this

public class TensorFlowActivity extends AppCompatActivity {



private static final String MODEL_PATH = "mobilenet_quant_v1_224.tflite";

private static final boolean QUANT = true;

private static final String LABEL_PATH = "labels.txt";

private static final int INPUT_SIZE = 224;



private Classifier classifier;



private Executor executor = Executors.newSingleThreadExecutor();

private TextView textViewResult;

private Button btnDetectObject, btnToggleCamera;

private ImageView imageViewResult;

private CameraView cameraView;



@Override

protected void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);

setContentView(R.layout.activity_tesnsorflow);

cameraView = findViewById(R.id.cameraView);

imageViewResult = findViewById(R.id.imageViewResult);

textViewResult = findViewById(R.id.textViewResult);

textViewResult.setMovementMethod(new ScrollingMovementMethod());



btnToggleCamera = findViewById(R.id.btnToggleCamera);

btnDetectObject = findViewById(R.id.btnDetectObject);



cameraView.addCameraKitListener(new CameraKitEventListener() {

@Override

public void onEvent(CameraKitEvent cameraKitEvent) {



}



@Override

public void onError(CameraKitError cameraKitError) {



}



@Override

public void onImage(CameraKitImage cameraKitImage) {



Bitmap bitmap = cameraKitImage.getBitmap();



bitmap = Bitmap.createScaledBitmap(bitmap, INPUT_SIZE, INPUT_SIZE, false);



imageViewResult.setImageBitmap(bitmap);



final List<Classifier.Recognition> results = classifier.recognizeImage(bitmap);



textViewResult.setText(results.toString());



}



@Override

public void onVideo(CameraKitVideo cameraKitVideo) {



}

});



btnToggleCamera.setOnClickListener(new View.OnClickListener() {

@Override

public void onClick(View v) {

cameraView.toggleFacing();

}

});



btnDetectObject.setOnClickListener(new View.OnClickListener() {

@Override

public void onClick(View v) {

cameraView.captureImage();

}

});



initTensorFlowAndLoadModel();

}



@Override

protected void onResume() {

super.onResume();

cameraView.start();

}



@Override

protected void onPause() {

cameraView.stop();

super.onPause();

}



@Override

protected void onDestroy() {

super.onDestroy();

executor.execute(new Runnable() {

@Override

public void run() {

classifier.close();

}

});

}



private void initTensorFlowAndLoadModel() {

executor.execute(new Runnable() {

@Override

public void run() {

try {

classifier = TensorFlowImageClassifier.create(

getAssets(),

MODEL_PATH,

LABEL_PATH,

INPUT_SIZE,

QUANT);

makeButtonVisible();

} catch (final Exception e) {

throw new RuntimeException("Error initializing TensorFlow!", e);

}

}

});

}



private void makeButtonVisible() {

runOnUiThread(new Runnable() {

@Override

public void run() {

btnDetectObject.setVisibility(View.VISIBLE);

}

});

}

}

Step 6 → Create a class TensorFlowImageClassifier.class for image processing and and get the information of images.

Note- To get the info about the images you just need to call run method of Interpeter and pass image data and the labels array .

interpreter.run(byteBuffer, result);

below the whole code is there.

public class TensorFlowImageClassifier implements Classifier {



private static final int MAX_RESULTS = 3;

private static final int BATCH_SIZE = 1;

private static final int PIXEL_SIZE = 3;

private static final float THRESHOLD = 0.1f;



private static final int IMAGE_MEAN = 128;

private static final float IMAGE_STD = 128.0f;



private Interpreter interpreter;

private int inputSize;

private List<String> labelList;

private boolean quant;



private TensorFlowImageClassifier() {



}



static Classifier create(AssetManager assetManager,

String modelPath,

String labelPath,

int inputSize,

boolean quant) throws IOException {



TensorFlowImageClassifier classifier = new TensorFlowImageClassifier();

classifier.interpreter = new Interpreter(classifier.loadModelFile(assetManager, modelPath), new Interpreter.Options());

classifier.labelList = classifier.loadLabelList(assetManager, labelPath);

classifier.inputSize = inputSize;

classifier.quant = quant;



return classifier;

}



@Override

public List<Recognition> recognizeImage(Bitmap bitmap) {

ByteBuffer byteBuffer = convertBitmapToByteBuffer(bitmap);

if (quant) {

byte[][] result = new byte[1][labelList.size()];

interpreter.run(byteBuffer, result);

return getSortedResultByte(result);

} else {

float[][] result = new float[1][labelList.size()];

interpreter.run(byteBuffer, result);

return getSortedResultFloat(result);

}



}



@Override

public void close() {

interpreter.close();

interpreter = null;

}



private MappedByteBuffer loadModelFile(AssetManager assetManager, String modelPath) throws IOException {

AssetFileDescriptor fileDescriptor = assetManager.openFd(modelPath);

FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor());

FileChannel fileChannel = inputStream.getChannel();

long startOffset = fileDescriptor.getStartOffset();

long declaredLength = fileDescriptor.getDeclaredLength();

return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);

}



private List<String> loadLabelList(AssetManager assetManager, String labelPath) throws IOException {

List<String> labelList = new ArrayList<>();

BufferedReader reader = new BufferedReader(new InputStreamReader(assetManager.open(labelPath)));

String line;

while ((line = reader.readLine()) != null) {

labelList.add(line);

}

reader.close();

return labelList;

}



private ByteBuffer convertBitmapToByteBuffer(Bitmap bitmap) {

ByteBuffer byteBuffer;



if (quant) {

byteBuffer = ByteBuffer.allocateDirect(BATCH_SIZE * inputSize * inputSize * PIXEL_SIZE);

} else {

byteBuffer = ByteBuffer.allocateDirect(4 * BATCH_SIZE * inputSize * inputSize * PIXEL_SIZE);

}



byteBuffer.order(ByteOrder.nativeOrder());

int[] intValues = new int[inputSize * inputSize];

bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());

int pixel = 0;

for (int i = 0; i < inputSize; ++i) {

for (int j = 0; j < inputSize; ++j) {

final int val = intValues[pixel++];

if (quant) {

byteBuffer.put((byte) ((val >> 16) & 0xFF));

byteBuffer.put((byte) ((val >> 8) & 0xFF));

byteBuffer.put((byte) (val & 0xFF));

} else {

byteBuffer.putFloat((((val >> 16) & 0xFF) - IMAGE_MEAN) / IMAGE_STD);

byteBuffer.putFloat((((val >> 8) & 0xFF) - IMAGE_MEAN) / IMAGE_STD);

byteBuffer.putFloat((((val) & 0xFF) - IMAGE_MEAN) / IMAGE_STD);

}



}

}

return byteBuffer;

}



@SuppressLint("DefaultLocale")

private List<Recognition> getSortedResultByte(byte[][] labelProbArray) {



PriorityQueue<Recognition> pq =

new PriorityQueue<>(

MAX_RESULTS,

new Comparator<Recognition>() {

@Override

public int compare(Recognition lhs, Recognition rhs) {

return Float.compare(rhs.getConfidence(), lhs.getConfidence());

}

});



for (int i = 0; i < labelList.size(); ++i) {

float confidence = (labelProbArray[0][i] & 0xff) / 255.0f;

if (confidence > THRESHOLD) {

pq.add(new Recognition("" + i,

labelList.size() > i ? labelList.get(i) : "unknown",

confidence, quant));

}

}



final ArrayList<Recognition> recognitions = new ArrayList<>();

int recognitionsSize = Math.min(pq.size(), MAX_RESULTS);

for (int i = 0; i < recognitionsSize; ++i) {

recognitions.add(pq.poll());

}



return recognitions;

}



@SuppressLint("DefaultLocale")

private List<Recognition> getSortedResultFloat(float[][] labelProbArray) {



PriorityQueue<Recognition> pq =

new PriorityQueue<>(

MAX_RESULTS,

new Comparator<Recognition>() {

@Override

public int compare(Recognition lhs, Recognition rhs) {

return Float.compare(rhs.getConfidence(), lhs.getConfidence());

}

});



for (int i = 0; i < labelList.size(); ++i) {

float confidence = labelProbArray[0][i];

if (confidence > THRESHOLD) {

pq.add(new Recognition("" + i,

labelList.size() > i ? labelList.get(i) : "unknown",

confidence, quant));

}

}



final ArrayList<Recognition> recognitions = new ArrayList<>();

int recognitionsSize = Math.min(pq.size(), MAX_RESULTS);

for (int i = 0; i < recognitionsSize; ++i) {

recognitions.add(pq.poll());

}



return recognitions;

}



}

Step 7 → Create interface Classifier.class for get the info about the images in your activity.

public interface Classifier {



class Recognition {

/**

* A unique identifier for what has been recognized. Specific to the class, not the instance of

* the object.

*/

private final String id;



/**

* Display name for the recognition.

*/

private final String title;



/**

* Whether or not the model features quantized or float weights.

*/

private final boolean quant;



/**

* A sortable score for how good the recognition is relative to others. Higher should be better.

*/

private final Float confidence;



public Recognition(

final String id, final String title, final Float confidence, final boolean quant) {

this.id = id;

this.title = title;

this.confidence = confidence;

this.quant = quant;

}



public String getId() {

return id;

}



public String getTitle() {

return title;

}



public Float getConfidence() {

return confidence;

}



@Override

public String toString() {

String resultString = "";

if (id != null) {

resultString += "[" + id + "] ";

}



if (title != null) {

resultString += title + " ";

}



if (confidence != null) {

resultString += String.format("(%.1f%%) ", confidence * 100.0f);

}



return resultString.trim();

}

}





List<Recognition> recognizeImage(Bitmap bitmap);



void close();

}

Conclusion

So these are the steps to implement TensorFlow’s lite in your android project.