Tokenizing a string in C++ - GeeksforGeeks

14 Dec, 2021
Tokenizing a string denotes splitting a string with respect to some delimiter(s). There are many ways to tokenize a string. In this article four of them are explained:
A stringstream associates a string object with a stream allowing you to read from the string as if it were a stream.
Below is the C++ implementation : 
C++
// Tokenizing a string using stringstream#include <bits/stdc++.h> using namespace std; int main(){         string line = "GeeksForGeeks is a must try";         // Vector of string to save tokens    vector <string> tokens;         // stringstream class check1    stringstream check1(line);         string intermediate;         // Tokenizing w.r.t. space ' '    while(getline(check1, intermediate, ' '))    {        tokens.push_back(intermediate);    }         // Printing the token vector    for(int i = 0; i < tokens.size(); i++)        cout << tokens[i] << '\n';}
GeeksForGeeks
is
a
must
try
// Splits str[] according to given delimiters.
// and returns next token. It needs to be called
// in a loop to get all tokens. It returns NULL
// when there are no more tokens.
char * strtok(char str[], const char *delims);
Below is the C++ implementation : 
C++
// C/C++ program for splitting a string// using strtok()#include <stdio.h>#include <string.h> int main(){    char str[] = "Geeks-for-Geeks";     // Returns first token    char *token = strtok(str, "-");     // Keep printing tokens while one of the    // delimiters present in str[].    while (token != NULL)    {        printf("%s\n", token);        token = strtok(NULL, "-");    }     return 0;}
Geeks
for
Geeks
 
C
// C code to demonstrate working of// strtok#include <string.h>#include <stdio.h> // Driver functionint main(){ // Declaration of string    char gfg[100] = " Geeks - for - geeks - Contribute";     // Declaration of delimiter    const char s[4] = "-";    char* tok;     // Use of strtok    // get first token    tok = strtok(gfg, s);     // Checks for delimiter    while (tok != 0) {        printf(" %s\n", tok);         // Use of strtok        // go through other tokens        tok = strtok(0, s);    }     return (0);}
  Geeks 
  for 
  geeks 
  Contribute
Just like strtok() function in C, strtok_r() does the same task of parsing a string into a sequence of tokens. strtok_r() is a reentrant version of strtok().
There are two ways we can call strtok_r() 
// The third argument saveptr is a pointer to a char * 
// variable that is used internally by strtok_r() in 
// order to maintain context between successive calls
// that parse the same string.
char *strtok_r(char *str, const char *delim, char **saveptr);
Below is a simple C++ program to show the use of strtok_r() : 
C++
// C/C++ program to demonstrate working of strtok_r()// by splitting string based on space character.#include<stdio.h>#include<string.h> int main(){    char str[] = "Geeks for Geeks";    char *token;    char *rest = str;     while ((token = strtok_r(rest, " ", &rest)))        printf("%s\n", token);     return(0);}
Geeks
for
Geeks
In this method the tokenization is done on the basis of regex matches. Better for use cases when multiple delimiters are needed.
Below is a simple C++ program to show the use of std::sregex_token_iterator:
C++
// CPP program for above approach#include <iostream>#include <regex>#include <string>#include <vector> /** * @brief Tokenize the given vector   according to the regex * and remove the empty tokens. * * @param str * @param re * @return std::vector<std::string> */std::vector<std::string> tokenize(                     const std::string str,                          const std::regex re){    std::sregex_token_iterator it{ str.begin(),                             str.end(), re, -1 };    std::vector<std::string> tokenized{ it, {} };     // Additional check to remove empty strings    tokenized.erase(        std::remove_if(tokenized.begin(),                            tokenized.end(),                       [](std::string const& s) {                           return s.size() == 0;                       }),        tokenized.end());     return tokenized;} // Driver Codeint main(){    const std::string str = "Break string                   a,spaces,and,commas";    const std::regex re(R"([\s|,]+)");       // Function Call    const std::vector<std::string> tokenized =                           tokenize(str, re);       for (std::string token : tokenized)        std::cout << token << std::endl;    return 0;}
Break
string
a
spaces
and
commas
mani00manu
akshaysingh98088
cpp-string
cpp-stringstream
C Language
C++
Strings
Strings
CPP
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Comments
Old Comments
Dynamic Memory Allocation in C using malloc(), calloc(), free() and realloc()
std::sort() in C++ STL
Bitwise Operators in C/C++
rand() and srand() in C/C++
Left Shift and Right Shift Operators in C/C++
Vector in C++ STL
Initialize a vector in C++ (6 different ways)
std::sort() in C++ STL
Bitwise Operators in C/C++
Socket Programming in C/C++