How to Label Image in Android using Firebase ML Kit? - GeeksforGeeks

24 Feb, 2021
We have seen many apps in Android in which we will detect the object present in the image whether it may be any object. In this article, we will take a look at the implementation of image labeling in Android using Firebase ML Kit. 
We will be building a simple application in which we will be capturing an image of any object and from that, we will detect the objects present inside our image with the accuracy level. A sample video is given below to get an idea about what we are going to do in this article. Note that we are going to implement this project using the Java language. 
Step 1: Create a New Project
To create a new project in Android Studio please refer to How to Create/Start a New Project in Android Studio. Note that select Java as the programming language.
Step 2: Connect your app to Firebase
After creating a new project in Android Studio connect your app to Firebase. For connecting your app to firebase. Navigate to Tools on the top bar. After that click on Firebase. A new window will open on the right side. Inside that window click on Firebase ML and then click on Use Firebase ML kit in Android. You can see the option below screenshot.  
After clicking on this option on the next screen click on Connect to Firebase option to connect your app to Firebase. 
Step 3: Adding dependency for language translation to build.gradle file
Navigate to the Gradle Scripts > build.gradle(Module:app) and add the below dependency in the dependencies section.   
// dependency for firebase core.
implementation’com.google.firebase:firebase-core:15.0.2′
// Firebase ML dependency
implementation ‘com.google.firebase:firebase-ml-vision:24.0.3’
implementation ‘com.google.firebase:firebase-ml-vision-image-label-model:20.0.1’
Step 4: Adding permissions to access the Internet and meta-data in your Android Apps AndroidManifest file  
Navigate to the app > AndroidManifest.xml file and add the below code to it. Comments are added in the code to get to know in more detail.  
XML
<!-- below line is use to add camera feature in our app --><uses-feature  android:name="android.hardware.camera"  android:required="true" />  <!-- permission for internet --><uses-permission android:name="android.permission.INTERNET" />
Add the below line inside your application tag. 
XML
<meta-data  android:name="com.google.firebase.ml.vision.DEPENDENCIES"  android:value="label" />
Below is the complete code for the AndroidManifest.xml file.
XML
<?xml version="1.0" encoding="utf-8"?><manifest xmlns:android="http://schemas.android.com/apk/res/android"    package="com.example.labelimage">      <!-- below line is use to add camera feature in our app -->    <uses-feature        android:name="android.hardware.camera"        android:required="true" />      <!-- permission for internet -->    <uses-permission android:name="android.permission.INTERNET" />      <application        android:allowBackup="true"        android:icon="@mipmap/ic_launcher"        android:label="@string/app_name"        android:roundIcon="@mipmap/ic_launcher_round"        android:supportsRtl="true"        android:theme="@style/Theme.LabelImage">        <activity android:name=".MainActivity">            <intent-filter>                <action android:name="android.intent.action.MAIN" />                  <category android:name="android.intent.category.LAUNCHER" />            </intent-filter>        </activity>          <meta-data            android:name="com.google.firebase.ml.vision.DEPENDENCIES"            android:value="label" />              </application>  </manifest>
Step 5: Working with the activity_main.xml file
Navigate to the app > res > layout > activity_main.xml and add the below code to that file. Below is the code for the activity_main.xml file. 
XML
<?xml version="1.0" encoding="utf-8"?><RelativeLayout     xmlns:android="http://schemas.android.com/apk/res/android"    xmlns:tools="http://schemas.android.com/tools"    android:layout_width="match_parent"    android:layout_height="match_parent"    tools:context=".MainActivity">      <!--image view for displaying our image-->    <ImageView        android:id="@+id/image"        android:layout_width="match_parent"        android:layout_height="300dp"        android:layout_alignParentTop="true"        android:layout_centerHorizontal="true"        android:layout_marginTop="10dp"        android:scaleType="centerCrop" />      <LinearLayout        android:id="@+id/idLLButtons"        android:layout_width="match_parent"        android:layout_height="wrap_content"        android:layout_below="@id/image"        android:orientation="horizontal">          <!--button for capturing our image-->        <Button            android:id="@+id/snapbtn"            android:layout_width="0dp"            android:layout_height="wrap_content"            android:layout_margin="5dp"            android:layout_marginTop="30dp"            android:layout_weight="1"            android:text="SNAP"            android:textSize="25dp"            android:textStyle="bold" />          <!--button for detecting the objects-->        <Button            android:id="@+id/labelBtn"            android:layout_width="0dp"            android:layout_height="wrap_content"            android:layout_margin="5dp"            android:layout_marginTop="30dp"            android:layout_weight="1"            android:text="Label"            android:textSize="25dp"            android:textStyle="bold" />      </LinearLayout>      <!--recycler view for displaying the list of objects-->    <androidx.recyclerview.widget.RecyclerView        android:id="@+id/idRVResults"        android:layout_width="match_parent"        android:layout_height="wrap_content"        android:layout_below="@id/idLLButtons" />  </RelativeLayout>
Step 6: Creating a modal class for storing our data
Navigate to the app > java > your app’s package name > Right-click on it > New > Java class and name your class as DataModal and add the below code to it. Comments are added inside the code to understand the code in more detail.
Java
public class DataModal {      // variables for our     // string and confidence.    private String result;    private float confidence;      // constructor    public DataModal(String result, float confidence) {        this.result = result;        this.confidence = confidence;    }      // getter and setter methods    public float getConfidence() {        return confidence;    }      public void setConfidence(float confidence) {        this.confidence = confidence;    }          public String getResult() {        return result;    }      public void setResult(String result) {        this.result = result;    }      }
Step 7: Creating a layout file for displaying our recycler view items
Navigate to the app > res > layout > Right-click on it > New > layout resource file and name it as result_rv_item and add below code to it. 
XML
<?xml version="1.0" encoding="utf-8"?><androidx.cardview.widget.CardView     xmlns:android="http://schemas.android.com/apk/res/android"    xmlns:app="http://schemas.android.com/apk/res-auto"    android:layout_width="match_parent"    android:layout_height="wrap_content"    android:layout_margin="5dp"    android:elevation="8dp"    app:cardCornerRadius="8dp">      <RelativeLayout        android:layout_width="match_parent"        android:layout_height="wrap_content"        android:layout_margin="3dp">          <!--text view for our result-->        <TextView            android:id="@+id/idTVResult"            android:layout_width="match_parent"            android:layout_height="wrap_content"            android:padding="3dp"            android:text="Result"            android:textColor="@color/black" />          <!--text view for our confidence-->        <TextView            android:id="@+id/idTVConfidence"            android:layout_width="match_parent"            android:layout_height="wrap_content"            android:layout_below="@id/idTVResult"            android:padding="3dp"            android:text="Confidence"            android:textColor="@color/black" />      </RelativeLayout>  </androidx.cardview.widget.CardView>
Step 8: Creating an adapter class for our RecyclerView
Navigate to the app > java > your app’s package name > Right-click on it > New > Java class and name it as resultRVAdapter and add the below code to it. Comments are added inside the code to understand the code in more detail.
Java
import android.content.Context;import android.view.LayoutInflater;import android.view.View;import android.view.ViewGroup;import android.widget.TextView;  import androidx.annotation.NonNull;import androidx.recyclerview.widget.RecyclerView;  import java.util.ArrayList;  public class resultRVAdapter extends RecyclerView.Adapter<resultRVAdapter.ViewHolder> {          // arraylist for storing our data and context    private ArrayList<DataModal> dataModalArrayList;    private Context context;      // constructor for our variables    public resultRVAdapter(ArrayList<DataModal> dataModalArrayList, Context context) {        this.dataModalArrayList = dataModalArrayList;        this.context = context;    }      @NonNull    @Override    public ViewHolder onCreateViewHolder(@NonNull ViewGroup parent, int viewType) {        // inside on create view holder method we are inflating our layout file which we created.        View view = LayoutInflater.from(parent.getContext()).inflate(R.layout.result_rv_item, parent, false);        return new ViewHolder(view);    }      @Override    public void onBindViewHolder(@NonNull ViewHolder holder, int position) {        // inside on bind view holder method we are setting        // data to each item of recycler view.        DataModal modal = dataModalArrayList.get(position);        holder.resultTV.setText(modal.getResult());        holder.confidenceTV.setText("" + modal.getConfidence());    }      @Override    public int getItemCount() {        // returning the size of array list.        return dataModalArrayList.size();    }      public class ViewHolder extends RecyclerView.ViewHolder {        // creating variables for our text view.        private TextView resultTV, confidenceTV;          public ViewHolder(@NonNull View itemView) {            super(itemView);            // initializing our views with their ids.            resultTV = itemView.findViewById(R.id.idTVResult);            confidenceTV = itemView.findViewById(R.id.idTVConfidence);        }    }}
Step 9: Working with the MainActivity.java file
Go to the MainActivity.java file and refer to the following code. Below is the code for the MainActivity.java file. Comments are added inside the code to understand the code in more detail.
Java
import android.content.Intent;import android.graphics.Bitmap;import android.os.Bundle;import android.provider.MediaStore;import android.view.View;import android.widget.Button;import android.widget.ImageView;import android.widget.Toast;  import androidx.annotation.NonNull;import androidx.appcompat.app.AppCompatActivity;import androidx.recyclerview.widget.LinearLayoutManager;import androidx.recyclerview.widget.RecyclerView;  import com.google.android.gms.tasks.OnFailureListener;import com.google.android.gms.tasks.OnSuccessListener;import com.google.firebase.ml.vision.FirebaseVision;import com.google.firebase.ml.vision.common.FirebaseVisionImage;import com.google.firebase.ml.vision.label.FirebaseVisionImageLabel;import com.google.firebase.ml.vision.label.FirebaseVisionImageLabeler;  import java.util.ArrayList;import java.util.List;  public class MainActivity extends AppCompatActivity {      // variables for our image view, image bitmap,    // buttons, recycler view, adapter and array list.    private ImageView img;    private Button snap, labelBtn;    private Bitmap imageBitmap;    private RecyclerView resultRV;    private resultRVAdapter resultRvAdapter;    private ArrayList<DataModal> dataModalArrayList;      @Override    protected void onCreate(Bundle savedInstanceState) {        super.onCreate(savedInstanceState);        setContentView(R.layout.activity_main);                  // initializing all our variables for views        img = (ImageView) findViewById(R.id.image);        snap = (Button) findViewById(R.id.snapbtn);        labelBtn = findViewById(R.id.labelBtn);        resultRV = findViewById(R.id.idRVResults);                  // initializing our array list        dataModalArrayList = new ArrayList<>();                  // initializing our adapter class.        resultRvAdapter = new resultRVAdapter(dataModalArrayList, MainActivity.this);                  // layout manager for our recycler view.        LinearLayoutManager manager = new LinearLayoutManager(MainActivity.this);                  // on below line we are setting layout manager         // and adapter to our recycler view.        resultRV.setLayoutManager(manager);        resultRV.setAdapter(resultRvAdapter);          // adding on click listener for our label button.        labelBtn.setOnClickListener(new View.OnClickListener() {            @Override            public void onClick(View v) {                // calling method                 // to label images.                labelImage();            }        });        // adding on click listener for our snap button.        snap.setOnClickListener(new View.OnClickListener() {            @Override            public void onClick(View v) {                // calling a method                 // to capture an image.                dispatchTakePictureIntent();            }        });    }      static final int REQUEST_IMAGE_CAPTURE = 1;      private void dispatchTakePictureIntent() {        // inside this method we are calling an implicit intent to capture an image.        Intent takePictureIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);        if (takePictureIntent.resolveActivity(getPackageManager()) != null) {            // calling a start activity for result when image is captured.            startActivityForResult(takePictureIntent, REQUEST_IMAGE_CAPTURE);        }    }      @Override    protected void onActivityResult(int requestCode, int resultCode, Intent data) {        super.onActivityResult(requestCode, resultCode, data);        // inside on activity result method we are setting        // our image to our image view from bitmap.        if (requestCode == REQUEST_IMAGE_CAPTURE && resultCode == RESULT_OK) {            Bundle extras = data.getExtras();            imageBitmap = (Bitmap) extras.get("data");            // on below line we are setting our            // bitmap to our image view.            img.setImageBitmap(imageBitmap);        }    }      private void labelImage() {                // inside the label image method we are calling a         // firebase vision image and passing our image bitmap to it.        FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(imageBitmap);                  // on below line we are creating a labeler for our image bitmap        // and creating a variable for our firebase vision image labeler.        FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance().getOnDeviceImageLabeler();                  // calling a method to process an image and adding on success listener method to it.        labeler.processImage(image).addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() {            @Override            public void onSuccess(List<FirebaseVisionImageLabel> firebaseVisionImageLabels) {                                    // inside on success method we are running a loop to get the data from our list.                for (FirebaseVisionImageLabel label : firebaseVisionImageLabels) {                                        // on below line we are getting text from our list.                    String text = label.getText();                                          // on below line we are getting its entity id                    String entityId = label.getEntityId();                                          // on below line we are getting the                    // confidence level of our modal.                    float confidence = label.getConfidence();                                         // after getting all data we are passing it to our array list.                    dataModalArrayList.add(new DataModal(text, confidence));                                          // after adding a new data we are notifying                     // our adapter that data has been updated.                    resultRvAdapter.notifyDataSetChanged();                }            }        }).addOnFailureListener(new OnFailureListener() {            @Override            public void onFailure(@NonNull Exception e) {                // error handling for on failure method                Toast.makeText(MainActivity.this, "Fail to get data..", Toast.LENGTH_SHORT).show();            }        });    }}
Now run your app and see the output of the app. 
Note: If you are facing the following NDK at ~/Library/Android/sdk/ndk-bundle did not have a source.properties file error then please refer to this article. 
Technical Scripter 2020
Android
Java
Machine Learning
Technical Scripter
Java
Android
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Resource Raw Folder in Android Studio
Flutter - Custom Bottom Navigation Bar
How to Read Data from SQLite Database in Android?
How to Post Data to API using Retrofit in Android?
Retrofit with Kotlin Coroutine in Android
Arrays in Java
Split() String method in Java with examples
For-each loop in Java
Object Oriented Programming (OOPs) Concept in Java
Arrays.sort() in Java with examples