Firebase MLKit

<h2>Introduction</h2>
<a src="https://firebase.google.com/products/ml-kit" target="_blank">ML Kit for Firebase</a> is a machine learning toolkit made by Google for Android and iOS. (Well still in Beta version)
With this kit you can use on-device pre-trained APIs :<ul><li>Text recognition</li><li>Face detection</li><li>Barcode scanning</li><li>Image labeling</li><li>Object detection & tracking</li><li>Language identification</li><li>Translation</li><li>Smart reply generator (only in english)</li></ul>You can also use cloud APIs for more accurate answer :
<ul><li>Text recognition</li><li>Image labeling</li><li>Landmark recognition</li></ul>You can use your custom pre-trained models and you can train your own classification model. (for images labeling only)
<h2>Android dependency</h2>
In app/build.gradle you should add :

```gradle
android {
  //...
  aaptOptions {
      noCompress "tflite"
  }
}  
dependencies {
  //...
  // ml-vision general
  implementation 'com.google.firebase:firebase-ml-vision:24.0.1'
  // Face Detection (contours)
  implementation 'com.google.firebase:firebase-ml-vision-face-model:19.0.0' 
  // Barcode Scanning
  implementation 'com.google.firebase:firebase-ml-vision-barcode-model:16.0.1'
  // Image labeling
  implementation 'com.google.firebase:firebase-ml-vision-image-label-model:19.0.0'
  // Object detection
  implementation 'com.google.firebase:firebase-ml-vision-object-detection-model:19.0.3'
  // ml-natural general
  implementation 'com.google.firebase:firebase-ml-natural-language:22.0.0'
  // Langauge identification
  implementation 'com.google.firebase:firebase-ml-natural-language-language-id-model:20.0.7'
  // Translation
  implementation 'com.google.firebase:firebase-ml-natural-language-translate-model:20.0.7'
  // Smart Replies
  implementation 'com.google.firebase:firebase-ml-natural-language-smart-reply-model:20.0.7'
}
apply plugin: 'com.google.gms.google-services'
```

If you want to use a custom pre-trained model(AutoML-trined model) to load your own model you will need to add : 
  implementation 'com.google.firebase:firebase-ml-vision-automl:18.0.3'
<h2>Text recognition</h2><img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/527965/201d9e31-d6c5-f1e1-97c7-4c16cd45acd8.png"/>

```java
// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Create an instance of FirebaseVisionTextRecognizer with on-device model
FirebaseVisionTextRecognizer detector = FirebaseVision.getInstance().getOnDeviceTextRecognizer();
// Or with cloud model
// FirebaseVisionTextRecognizer detector = FirebaseVision.getInstance().getCloudTextRecognizer();

// Process the image
Task<FirebaseVisionText> result =
        detector.processImage(image)
                .addOnSuccessListener(new OnSuccessListener<FirebaseVisionText>() {
                    @Override
                    public void onSuccess(FirebaseVisionText firebaseVisionText) {
                        // Task completed successfully
                        // ...
                    }
                })
                .addOnFailureListener(
                        new OnFailureListener() {
                            @Override
                            public void onFailure(@NonNull Exception e) {
                                // Task failed with an exception
                                // ...
                            }
                        });
```

FirebaseVisionText will contain bounding box, text, language recognized, paragraph, confidence score.<h2>Face detection</h2><img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/527965/18a30632-c2d1-e069-48f9-ede5e93d9de1.png" width="400px"> <img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/527965/fba831c8-a60b-5fd1-e058-00de898dbe3c.png" width="200px">
Face detection is done on device only and you can get facial contours too (optional).

```java
// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Set Options
FirebaseVisionFaceDetectorOptions options =
          new FirebaseVisionFaceDetectorOptions.Builder()
                .setPerformanceMode(FirebaseVisionFaceDetectorOptions.ACCURATE)
                .setClassificationMode(FirebaseVisionFaceDetectorOptions.ALL_CLASSIFICATIONS)
                .setLandmarkMode(FirebaseVisionFaceDetectorOptions.ALL_LANDMARKS)                                                                                
                .setContourMode(FirebaseVisionFaceDetectorOptions.ALL_CONTOURS)
                .build();
               
// Create an instance of FirebaseVisionFaceDetector
FirebaseVisionFaceDetector detector = FirebaseVision.getInstance().getVisionFaceDetector(options);

// Process the image
Task<List<FirebaseVisionFace>> result =
        detector.detectInImage(image)
                .addOnSuccessListener(
                        new OnSuccessListener<List<FirebaseVisionFace>>() {
                            @Override
                            public void onSuccess(List<FirebaseVisionFace> faces) {
                                // Task completed successfully
                                // ...
                            }
                        })
                .addOnFailureListener(
                        new OnFailureListener() {
                            @Override
                            public void onFailure(@NonNull Exception e) {
                                // Task failed with an exception
                                // ...
                            }
                        });

```
As a result FirebaseVisionFace contains : Face bounds, head rotation, eyes ears, mouth nose coordinate, classification probability (smiling, eyes opened, happy ...)
You can also get a tracking Id in case of video streaming.<h2>Barcode scanning</h2><img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/527965/4b2f6695-7000-290c-af11-9a44a649929e.png" width="300px">
Many different formats are supported :
Code 128, Code 39, Code 93, Codabar, EAN-13, EAN-8, ITF, UPC-A, UPC-E, QR Code, PDF417, Aztec, Data Matrix.

```java
// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Set Options
FirebaseVisionBarcodeDetectorOptions options =
        new FirebaseVisionBarcodeDetectorOptions.Builder()
        .setBarcodeFormats(
                FirebaseVisionBarcode.FORMAT_QR_CODE,
                FirebaseVisionBarcode.FORMAT_AZTEC)
        .build();

// Create an instance of FirebaseVisionBarcodeDetector
FirebaseVisionBarcodeDetector detector = FirebaseVision.getInstance().getVisionBarcodeDetector();

// Process the image
Task<List<FirebaseVisionBarcode>> result = detector.detectInImage(image)
        .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionBarcode>>() {
            @Override
            public void onSuccess(List<FirebaseVisionBarcode> barcodes) {
                // Task completed successfully
                // ...
            }
        })
        .addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
                // Task failed with an exception
                // ...
            }
                });
```
The results will depend on the barcode type
<h2>Image labeling</h2><img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/527965/dd7dbddf-52a5-71d0-da83-5eb78976f45d.png" width="300px">
Image labeling can be used on-device with ~400 labels or on-cloud with ~10000 labels

```java
// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Create an instance of FirebaseVisionImageLabeler with on-device model
FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance().getOnDeviceImageLabeler();
// Or with cloud model
// FirebaseVisionCloudImageLabelerOptions options = new FirebaseVisionCloudImageLabelerOptions.Builder().setConfidenceThreshold(0.7f).build();
// FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance().getOnDeviceImageLabeler(options);

// Process the image
labeler.processImage(image)
    .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() {
      @Override
      public void onSuccess(List<FirebaseVisionImageLabel> labels) {
        // Task completed successfully
        // ...
      }
    })
    .addOnFailureListener(new OnFailureListener() {
      @Override
      public void onFailure(@NonNull Exception e) {
        // Task failed with an exception
        // ...
      }
    });
```
Return is just a list of FirebaseVisionImageLabel who contains : label and confidence score.
You can use AutoML Vision Edge to use your own model of classification.
<h2>Object detection & tracking</h2><img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/527965/66e757ca-af7f-b4e0-3c62-9b7437a4dd07.png" width="300px">
With this you can identify main object and track it (when streaming)

```java
// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Multiple object detection in static images
FirebaseVisionObjectDetectorOptions options =
        new FirebaseVisionObjectDetectorOptions.Builder()
                .setDetectorMode(FirebaseVisionObjectDetectorOptions.SINGLE_IMAGE_MODE)
                .enableMultipleObjects()
                .enableClassification()  // Optional
                .build();

// Create an instance of FirebaseVisionObjectDetector
FirebaseVisionObjectDetector objectDetector = FirebaseVision.getInstance().getOnDeviceObjectDetector(options);

// Process the image
objectDetector.processImage(image)
        .addOnSuccessListener(
                new OnSuccessListener<List<FirebaseVisionObject>>() {
                    @Override
                    public void onSuccess(List<FirebaseVisionObject> detectedObjects) {
                        // Task completed successfully
                        // ...
                    }
                })
        .addOnFailureListener(
                new OnFailureListener() {
                    @Override
                    public void onFailure(@NonNull Exception e) {
                        // Task failed with an exception
                        // ...
                    }
                });
```
Result is a list of FirebaseVisionObject who contains: tracking Id, bounds, category, confidence score.
<h2>Landmark recognition</h2><img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/527965/83447342-1489-2acc-3b0a-a6b3cde62d86.png" width="600px">
You can recognize well-known landmarks in an image. This api can only be use on-cloud.

```java

// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Set the options
FirebaseVisionCloudDetectorOptions options =
        new FirebaseVisionCloudDetectorOptions.Builder()
                .setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL)
                .setMaxResults(15)
                .build();

// Create an instance of FirebaseVisionCloudLandmarkDetector
FirebaseVisionCloudLandmarkDetector detector = FirebaseVision.getInstance().getVisionCloudLandmarkDetector(options);

// Process the image
Task<List<FirebaseVisionCloudLandmark>> result = detector.detectInImage(image)
        .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionCloudLandmark>>() {
            @Override
            public void onSuccess(List<FirebaseVisionCloudLandmark> firebaseVisionCloudLandmarks) {
                // Task completed successfully
                // ...
            }
        })
        .addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
                // Task failed with an exception
                // ...
            }
        });
```

Result is a list of FirebaseVisionCloudLandmark who contains: Name, bounds, latitude, longitude, confidence score.
<h2>Language identification</h2>
This api doesn't use image but String.

```java
FirebaseLanguageIdentification languageIdentifier =
        FirebaseNaturalLanguage.getInstance().getLanguageIdentification();
languageIdentifier.identifyAllLanguages(text)
      .addOnSuccessListener(
          new OnSuccessListener<String>() {
            @Override
            public void onSuccess(List<IdentifiedLanguage> identifiedLanguages) {
              for (IdentifiedLanguage identifiedLanguage : identifiedLanguages) {
                String language = identifiedLanguage.getLanguageCode();
                float confidence = identifiedLanguage.getConfidence();
                Log.i(TAG, language + " (" + confidence + ")");
              }
            }
          })
      .addOnFailureListener(
          new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
              // Model couldn’t be loaded or other internal error.
              // ...
            }
          });
```
Result is a list of IdentifiedLanguage who contains: Language Code and confidence score.
<h2>Translation</h2>Translation can be done with a on-device api but it can be use only for casual and simple translation over 59 languages (Japanese is supported). 
Model is trained to translate to and from English; so if you choose to translate between non-English languages, English will be used as an intermediate translation, which can affect quality.

```java
// Create an English-Japanese translator:
FirebaseTranslatorOptions options =
        new FirebaseTranslatorOptions.Builder()
                .setSourceLanguage(FirebaseTranslateLanguage.EN)
                .setTargetLanguage(FirebaseTranslateLanguage.JP)
                .build();
final FirebaseTranslator englishJapaneseTranslator =
        FirebaseNaturalLanguage.getInstance().getTranslator(options);

final String text = "Merry Christmas";

// We need to download the model first
// Each model is around 30MB and are stored locally to be reused
FirebaseModelDownloadConditions conditions = new FirebaseModelDownloadConditions.Builder()
    .requireWifi()
    .build();
englishJapaneseTranslator.downloadModelIfNeeded(conditions)
      .addOnSuccessListener(
          new OnSuccessListener<Void>() {
            @Override
            public void onSuccess(Void v) {
              // Model downloaded successfully. We can start translation
              englishJapaneseTranslator.translate(text)
                  .addOnSuccessListener(
                      new OnSuccessListener<String>() {
                         @Override
                         public void onSuccess(@NonNull String translatedText) {
                            // Translation successful.
                            // translatedText <- "メリークリスマス"
                         }
                  })
                  .addOnFailureListener(
                      new OnFailureListener() {
                          @Override
                          public void onFailure(@NonNull Exception e) {
                             // Error during the translation.
                          }
                      });
            }
          })
      .addOnFailureListener(
          new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
              // Model couldn’t be downloaded or other internal error.
            }
          });
```
<h2>Smart reply generator (only in english)</h2>
The model work with the 10 most recent messages and provides a maximum of 3 suggested responses.

```java
// Define a conversation history
// Local User speaks to Remote User
// Smart reply is about what Local User may answer.
List<FirebaseTextMessage> conversation = new ArrayList<>();
conversation.add(FirebaseTextMessage.createForRemoteUser("It's Christmas time", System.currentTimeMillis(),"userId1"));
conversation.add(FirebaseTextMessage.createForLocalUser("Kids are happy", System.currentTimeMillis()));
conversation.add(FirebaseTextMessage.createForRemoteUser("Will Santa Claus come tonight ?", System.currentTimeMillis(),"userId1"));

FirebaseSmartReply smartReply = FirebaseNaturalLanguage.getInstance().getSmartReply();
smartReply.suggestReplies(conversation)
        .addOnSuccessListener(new OnSuccessListener<SmartReplySuggestionResult>() {
            @Override
            public void onSuccess(SmartReplySuggestionResult result) {
                if (result.getStatus() == SmartReplySuggestionResult.STATUS_NOT_SUPPORTED_LANGUAGE) {
                    // The conversation's language isn't supported, so the
                    // the result doesn't contain any suggestions.
                } else if (result.getStatus() == SmartReplySuggestionResult.STATUS_SUCCESS) {
                    // Task completed successfully
                    for (SmartReplySuggestion suggestion : result.getSuggestions()) {
                        String replyText = suggestion.getText();
                    }
                }
            }
        })
        .addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
                // Task failed with an exception
            }
        });
```
As result a list of SmartReplySuggestion Object. Each one will contain only a text.
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/527965/c84bbaed-7ae1-e70e-65c2-812f8942b8cc.png)