Q:

Pattern match string from column in spark dataframe

I have column in spark dataframe, where i need to search the data with only string containing "xyz" and to stored in new column. 
Input   (need the only field from column having xyz )
col A      colB                                                        
A          bid:76563,bid:76589,bid:76591,ms:ms15-097,xyz:3089656      
B          xyz:4462915,xyz:4462917,xyz:4462918                        

Required Output
col A      colB                                                        colC
A          bid:76563,bid:76589,bid:76591,ms:ms15-097,xyz:3089656      xyz:3089656
B          xyz:4462915,xyz:4462917,xyz:4462918                        xyz:4462915,xyz:4462917,xyz:4462918

I have 100k rows and cannot used groupby on colA  using collect_list, can you please to get the required output. 

A:

If you are using Spark 2.4+ then you can split the colB with comma , and use built in functions as expressions 
import org.apache.spark.sql.functions._ 

import spark.implicits._ 
val df = Seq(
  ("A", "bid:76563,bid:76589,bid:76591,ms:ms15-097,xyz:3089656"),
  ("B", "xyz:4462915,xyz:4462917,xyz:4462918")
).toDF("colA", "colB")

val newDF = df.withColumn("split", split($"colB", ","))
  .selectExpr("*", "filter(split, x -> x LIKE 'xyz%' ) filteredB")
  .withColumn("colC", concat_ws(",", $"filteredB"))
  .drop("split", "filteredB")

newDF.show(false)

Output:
+----+-----------------------------------------------------+-----------------------------------+
|colA|colB                                                 |colC                               |
+----+-----------------------------------------------------+-----------------------------------+
|A   |bid:76563,bid:76589,bid:76591,ms:ms15-097,xyz:3089656|xyz:3089656                        |
|B   |xyz:4462915,xyz:4462917,xyz:4462918                  |xyz:4462915,xyz:4462917,xyz:4462918|
+----+-----------------------------------------------------+-----------------------------------+

