Lucene - StandardAnalyzer

This is the most sophisticated analyzer and is capable of handling names, email addresses, etc. It lowercases each token and removes common words and punctuations, if any.
Following is the declaration for the org.apache.lucene.analysis.StandardAnalyzer class −
public final class StandardAnalyzer
   extends StopwordAnalyzerBase

Following are the fields for the org.apache.lucene.analysis.StandardAnalyzer class −
static int DEFAULT_MAX_TOKEN_LENGTH – This is the default maximum allowed token length.
static int DEFAULT_MAX_TOKEN_LENGTH – This is the default maximum allowed token length.
static Set<?> STOP_WORDS_SET - An unmodifiable set containing some common English words that are usually not useful for searching.
static Set<?> STOP_WORDS_SET - An unmodifiable set containing some common English words that are usually not useful for searching.
The following table shows the different class constructors −
StandardAnalyzer(Version matchVersion)
Builds an analyzer with the default stop words (STOP_WORDS_SET).
StandardAnalyzer(Version matchVersion, File stopwords)
Deprecated. Use StandardAnalyzer(Version, Reader) instead.
StandardAnalyzer(Version matchVersion, Reader stopwords)
Builds an analyzer with the stop words from the given reader.
StandardAnalyzer(Version matchVersion, Set<?> stopWords) 
Builds an analyzer with the given stop words.
The following table shows the different class methods −
protected Reusable Analyzer Base. Token Stream Components create Components(String fieldName, Reader reader)
Creates a new ReusableAnalyzerBase.TokenStreamComponents instance for this analyzer.
int getMaxTokenLength()
void setMaxTokenLength(int length)
Sets maximum allowed token length.
This class inherits methods from the following classes −
org.apache.lucene.analysis.StopwordAnalyzerBase
org.apache.lucene.analysis.ReusableAnalyzerBase
org.apache.lucene.analysis.Analyzer
java.lang.Object
private void displayTokenUsingStandardAnalyzer() throws IOException {
   String text 
      = "Lucene is simple yet powerful java based search library.";
   Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_36);
   TokenStream tokenStream 
      = analyzer.tokenStream(LuceneConstants.CONTENTS,
        new StringReader(text));
   TermAttribute term = tokenStream.addAttribute(TermAttribute.class);
   
   while(tokenStream.incrementToken()) {
      System.out.print("[" + term.term() + "] ");
   }
}
Let us create a test Lucene application to test search using BooleanQuery.
Create a project with a name LuceneFirstApplication under a package com.tutorialspoint.lucene as explained in the Lucene - First Application chapter. You can also use the project created in Lucene - First Application chapter as such for this chapter to the understand searching process.
Create LuceneConstants.java as explained in the Lucene - First Application chapter. Keep the rest of the files unchanged.
Create LuceneTester.java as mentioned below.
Clean and Build the application to make sure the business logic is working as per the requirements.
This class is used to provide various constants to be used across the sample application.
package com.tutorialspoint.lucene;

public class LuceneConstants {
   public static final String CONTENTS = "contents";
   public static final String FILE_NAME = "filename";
   public static final String FILE_PATH = "filepath";
   public static final int MAX_SEARCH = 10;
}
This class is used to test the searching capability of the Lucene library.
package com.tutorialspoint.lucene;

import java.io.IOException;
import java.io.StringReader;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.StandardAnalyzer;
import org.apache.lucene.analysis.tokenattributes.TermAttribute;
import org.apache.lucene.util.Version;

public class LuceneTester {
	
   public static void main(String[] args) {
      LuceneTester tester;

      tester = new LuceneTester();
   
      try {
         tester.displayTokenUsingStandardAnalyzer();
      } catch (IOException e) {
         e.printStackTrace();
      }
   }

   private void displayTokenUsingStandardAnalyzer() throws IOException {
      String text 
         = "Lucene is simple yet powerful java based search library.";
      Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_36);
      TokenStream tokenStream = analyzer.tokenStream(
         LuceneConstants.CONTENTS, new StringReader(text));
      TermAttribute term = tokenStream.addAttribute(TermAttribute.class);
      while(tokenStream.incrementToken()) {
         System.out.print("[" + term.term() + "] ");
      }
   }
}
Once you are done with the creation of the source, you can proceed by compiling and running your program. To do this, keep the LuceneTester.Java file tab active and use either the Run option available in the Eclipse IDE or use Ctrl + F11 to compile and run your LuceneTester application. If your application runs successfully, it will print the following message in Eclipse IDE's console −
[lucene] [simple] [yet] [powerful] [java] [based] [search] [library] 

 Print
 Add Notes
 Bookmark this page