AI-Mock-Interviewer/PDF_Extracted_Data huggingface dataset:
Question: how to access h base tables from hive?
Answer: Using Hive -HBase storage handler, you can access the HBase tables from Hive and once you are connected, you can query HBase using the SQL queries from Hive. You can also join multiple tables in HBase from Hive and re trieve the result.
Question: when running a join query i see outofmemoryerror s?
Question: communications exception communications link failure?
Question: does hive support unicode?
Question: are hive sql identifiers eg table names columns etc case sensitive?
Answer: No, Hive is case insensitive.
Question: what is the best way to loadxml data into hive?
Answer: The easiest way is to use the Hive XML SerDe (com.ibm.spss.hive.serde2.xml.XmlSerDe), which will all ow you to directly import and work with XML data.
Question: when hive is not suitable?
Question: mention what are the different modes of hive?
Answer: Depending on the size of data nodes in Hadoop, Hive can operate in two modes. These modes are, Local mode and Map reduce mode
Question: mention what is hs2hiveserver2?
Question: mention what hive query processor does?
Answer: Hive query processor convert graph of MapReduce jobs with the execution time framework. So that the jobs can be executed in the order of dependencies. Follow  Me : https://www.yo utube.com/c/SauravAgarwa l
Question: mention what are the steps of hive in query processor?
Answer: The components of a Hive query processor include, 1. Logical Plan Generation 2. Physical Plan Generation Execution Engine 3. Operators 4. UDFs and UDAFs 5. Optimizer 6. Parser 7. Semantic Analyzer 8. Type Checking
Question: explain how can you change a column datatype in hive?
Answer: You can change a column data type in Hive by using command, ALTER TABLE table_name CHANGE column_name column_name new_datatype;
Question: mention what is the difference between order by and sortby in hive?
Answer: SORT BY will sort the data within each reducer. You can use any number of reducers for SORT BY operation. ORDER BY will sort all of the data together, which has to pass through one reduc er. Thus, ORDER BY in hive uses a single.
Question: explain when to use explode in hive?
Answer: Follow  Me : https://www.yo utube.com/c/SauravAgarwa l Hadoop developers sometimes take an array as input and convert into a separate table row. To convert complex data types into desired table formats, then we can use explode  function.
Question: mention how can you stop a partition form being queried?
Question: can were name a hive table?
Answer: yes, using below command  Alter Table table_name RENAME TO new_name
Question: what is the default location where hive stores table data?
Answer: hdfs://namenode_server/user/hive/warehouse
Question: is there a date datatype in hive?
Question: can we run unix shell commands from hive give example
Answer: Yes, using the ! mark just before the command. For example !pwd at hive prompt will list the current directory.
Question: can hive queries be executed from script file show?
Answer: Using the source comm and. Example  Hive> source /path/to/file/file_with_query.hql
Question: what is the importance of hive rcfile?
Question: what are the default record and field delimiter used for hive text files?
Answer: The default record delimiter is  \n And the filed delimiters are  \001,\002,\003
Question: what do you mean by schema on read?
Answer: The schema is validated with the data when reading the data and not enforced when writing data.
Question: how do you list all databases whose name starts with p?
Question: what does the use command in hive do?
Answer: With the use command you fix the database on which all the subsequent hive queries will run.
Question: how can you delete the db property in hive?
Answer: There is no way you can delete the DBPROPERTY.
Question: what is the significance of the line?
Question: how do you check if a particular partition exists?
Question: which java class handles the input and output records encoding into files in hive tables?
Question: what is the significance of if exists clause while dropping a table?
Question: when you point a partition of a hive table to a new directory what happens to the data?
Question: does the archiving of hive tables it saves any spaces in hdfs?
Question: a hdfs file and not a local file?
Question: are new and files which already exist?
Question: what is a table generating function on hive?
Answer: A table generating function is a function which takes a single column as argument and expands it to multiple column or rows. Example exploe().
Question: how can hive avoid map reduce?
Question: what is the difference between like an dr like operators in hive?
Question: is it possible to create cartesian join between 2 tables using hive?
Answer: No. As this kind of Join can not be implemented in map reduce
Question: follow me https www you tubecomcsauravagarwal70 what should be the order of table size in a join query?
Question: what is the usefulness of the distributed by clause in hive?
Question: how will you convert the string 512toa floatvalue in the price column?
Answer: Select cast(price as FLOAT)
Question: what will be the result when you do cast abc as int?
Answer: Hive will return NULL
Question: can we load data into a view?
Question: what types of costs are associated in creating index on hive tables?
Question: what does stream table tablename do?
Question: can a partition be archived what are the advantages and disadvantages?
Answer: Follow  Me : https://www.yo utube.com/c/SauravAgarwa l Yes. A partition can be archived. Advantage is it decreases the number of files stored in namenode and the archived file can be queried using hive. The disadvantage is it will cause less efficient query and does not offer any space savings.
Question: what is a generic udf in hive?
Question: the following statement failed to execute what can be the cause?
Question: how do you specify the table creator name when creating a table in hive?
Question: which method has to be overridden when we use custom udf in hive?
Question: same time?
Answer: Follow  Me : https://www.yo utube.com/c/SauravAgarwa l The default metastore configuration allows only one Hive session to be opened at a time for accessing the metastore. Therefore, if multiple clients try to access the metastore at the same time, they will get an error. One has to use a standalone metastore, i.e. Local or remote metastore configuration in Apache Hive for allowing access to multiple clients concurrently. Following are the steps to configure MySQL database as the local metastor e in Apache Hive: One should make the following changes in hive -site.xml: 1. javax.jdo.option.ConnectionURL property should be set to
Question: jdbc mysql host dbname create database if not exist true
Answer: 2. javax.jdo.option.ConnectionDriverName property should be set to com. mysql.jdbc.Driver. One should also set the username and password as: 3. javax.jdo.option.ConnectionUserName is set to desired username. 4. javax.jdo.option.ConnectionPassword is set to the desired password. The JDBC driver JAR file for MySQL must be on the Hive classpath, i.e. The jar file should be copied into the Hive lib directory. Now, after restarting the Hive shell, it will automatically connect to the MySQL database which is running as a standalone metastore.
Question: is it possible to change the default location of a managed table?
Question: when should we use sort by instead of order by?
Answer: We should use SORT BY instead of ORDER BY whe n we have to sort huge datasets because SORT BY clause sorts the data using multiple reducers whereas ORDER BY sorts all of the data together using a single reducer. Therefore, using ORDER BY against a large number of inputs will take a lot of time to exec ute.
Question: what is dynamic partitioning and when is it used?
Question: order to do so?
Question: how can you add a new partition for the month december in the above partitioned table?
Question: what is the default maximum dynamic partition that can be created by a mapper reducer?
Question: how can you change it?
Question: requires at least one static partition column how will you remove this error?
Question: how will you consume this csv file into the hive warehouse using built ser de?
Question: files without degrading the performance of the system?
Question: can we change settings within hive session if yes how?
Question: is it possible to add 100nodeswhenwehave100 nodes already in hive how?
Question: explain the concatenation function in hive with an example?
Question: explain trim and reverse function in hive with examples?
Question: explain process to access subdirectories recursively in hive queries?
Answer: By using below commands we can access sub directories recursively in Hive hive> Set mapred.input.dir.recursive=true; hive> Set hive.mapred.supports.subdirectories=true; Hive tables can be pointed to the h igher level directory and this is suitable for the directory structure which is like /data/country/state/city/
Question: how to skip header rows from a table in hive?
Question: what is the maximum size of string datatype supported by hive mention the hive support
Question: binary formats?
Question: what is the precedence order of hive configuration?
Question: if you run a select query in hive why does it not run map reduce?
Question: how hive can improve performance with orc format tables?
Question: explain about the different types of join in hive?
Question: how can you configure remote meta store mode in hive?
Question: what happens on executing the below query after executing the below query if you modify the
Question: column how will the changes be tracked?
Question: how to load data from a txt file to table stored as orc in hive?
Question: hive?
Question: how to improve hive query performance with had oop?
Question: how do i query from a horizontal output to vertical output?
Question: and ve numbers?
Answer: we can try to use regexp_extract instead: regexp_extract('abcd -9090','.*( -[0-9]+)',1)
Question: follow me https www you tubecomcsauravagarwal111 what is hive tablename maximum character limit?
Question: hive?
Answer: use from_unixtime in conjunction with unix_timestamp. select from_unixtime(unix_timestamp(`date`,'MMM dd, yyyy'),'yyyy -MM-dd')
Question: how to drop the hive database whether it contains some tables?
Answer: Use cascade command while drop the database. Example: hive> drop database sampleDB cascade;
Question: i dropped and recreated hive external table but no data shown so what should i do?
Question: difference between rdd data frame dataset?
Question: when to user dds?
Answer: Consider these scenarios or common use cases for using RDDs when: 1. you want low -level transformation and actions and control on your dataset; your data is unstructured, such as media streams or streams of text; 2. you want to manipulate your data with functional programming constructs than domain specific  expressions; 3. you do nt care about imposing a schema, such as columnar format, while processing or accessing data attributes by name or column; and 4. you can forgo some optimization and performance benefits available with DataFrames and Datasets for structured and semi -structur ed data.
Question: what are the various modes in which spark runs on yarn client vs cluster mode
Answer: YARN client mode: The driver runs on the machine from which client is connected YARN Cluster Mode: The driver runs inside cluster
Question: what is dag directed acyclic graph?
Question: what is a rdd and how it works internally?
Question: what do we mean by partitions or slices?
Question: what is the difference between map and flat map?
Question: how can you minimize data transfers when working with spark?
Answer: The various ways in which data transfers can be minimized when working  with Apache Spark are: 1.Broadcast Variable - Broadcast variable enhances the efficiency of joins between small and large RDDs. 2.Accumulators - Accumulators help update the values of variables in parallel while executing. 3.The most common way is to avoid  operations ByKey, repartition or any other operations which trigger shuffles.
Question: why is there a need for broadcast variables when working with apache spark?
Answer: These are read only variables, present in -memory cache on every machine. When working with Spark , usage of broadcast variables eliminates the necessity to ship copies of a variable for every task, so data can be processed faster. Broadcast variables help in storing a lookup table inside the memory which enhances the retrieval efficiency when compared  to an RDD lookup ().
Question: how can you trigger automatic cleanups in spark to handle accumulated metadata?
Answer: You can trigger the clean -ups by setting the parameter "spark.cleaner.ttl" or by dividing the long running jobs into different batches and writing the intermediary results to the disk. Follow  Me : https://www.yo utube.com/c/SauravAgarwa l
Question: why is blink db used?
Answer: BlinkDB is a query engine for executing interactive SQL queries on huge volumes of data and renders query results marked with meaningful error bars. BlinkDB helps users balance query a ccuracy with response time.
Question: what is sliding window operation?
Answer: Sliding Window controls transmission of data packets between various computer networks. Spark Streaming library provides windowed computations where the transformations on RDDs are applied over a sliding window of data. Whenever the window slides, the RDDs that fall within the particular window are combined and operated upon to produce new RDDs of the windowed DStream.
Question: what is catalyst optimiser?
Question: what do you understand by pair rdd?
Question: what is the difference between persist and cache?
Answer: persist  ()   allows   t he   user   to   specify   the   storage   level   whereas   cache  ()   uses   the default   storage level(MEMORY_ONLY).
Question: what are the various levels of persistence in apache spark?
Question: does apache spark provide checkpointing?
Question: what do you understand by lazy evaluation?
Answer: Spark is intellectual in the manner in which it operates on data. When you tell Spark to operate  on a given dataset, it heeds the instructions and makes a note of it, so that it does not forget - but it does nothing, unless asked for the final result. When a transformation like map () is called on a RDD -the operation is not performed  immediately. Transformations in Spark are not evaluated till you perform an action. This helps optimize the overall data processing workflow.
Question: what do you understand by schema rdd?
Answer: An RDD that consists of row objects (wrappers around basic string or integer arrays) w ith schema information about the type of data in each column. Dataframe is an example of SchemaRDD.
Question: what are the disadvantages of using apache spark over had oop map reduce?
Answer: Apache spark does not scale well for compute intensive jobs and consumes large n umber of system resources. Apache Sparks in -memory capability at times comes a major roadblock for cost efficient processing of big data. Also, Spark does have its own file management system and hence needs to be integrated with other cloud based data pla tforms or apache hadoop.
Question: what is lineage graph in spark?
Question: what do you understand by executor memory in a spark application?
Answer: Follow  Me : https://www.yo utube.com/c/SauravAgarwa l Every spark application has same fixed heap size and fixed number of cores for a spark executor. The heap size is what referred to as the Spark executor memory which is controlled with the spark. executor.memory property of the -executor -memory flag. Every spark application will have one executor on each worker node. The executor  memory is basically a measure on how much memory of the worker node will the application utilize.
Question: what is an accumulator?
Answer: Accumulators are Sparks offline debuggers. Similar to Hadoop Counters, Accumulators provide the number of events in a  program. Accumulators are the variables that can be added through associative operations. Spark natively supports accumulators   of   numeric   value   types   and   standard   mutable   collections. AggregrateByKey()   and combineByKey() uses accumul ators.
Question: what is spark context?
Question: what is spark session?
Question: why rdd is an immutable?
Question: what is partitioner?
Question: what are the benefits of data frames?
Question: what is dataset?
Question: what are the benefits of data sets?
Question: what is shared variable in apache spark?
Answer: Shared variables are nothing but the variables that can be used in parallel operations. Follow  Me : https://www.yo utube.com/c/SauravAgarwa l Spark supports two types of shared variables: broadcast variables, which can be used to cache a value in memory on all nodes, and accumulators, which are variables that are only added to, such as counters and sums.
Question: how to accumulated metadata in apache spark?
Question: what is the difference between dsm and rdd?
Question: what is speculative execution in spark and how to enable it?
Answer: One more point is, Speculative execution will not stop the slow running task but it launch the new task in parallel. Tabular Form : Spark Property >> Default Value >> Description spark.speculation >> false >> enables ( true ) or disables ( false ) speculative execution of tasks. spark.speculation.interval >> 100ms >> The time interval to use before checking for speculat ive tasks. spark.speculation.multiplier >> 1.5 >> How many times slower a task is than the median to be for speculation. spark.speculation.quantile >> 0.75 >> The percentage of tasks that has not finished yet at which to start speculation.
Question: how is fault tolerance achieved in apache spark?
Question: combine by key?
Question: explain the map partitions and map partitions with index?
Question: explain fold operation in spark?
Question: difference between text file vs whole text file?
Question: what is co group operation?
Question: explain pipe operation?
Question: explain coalesce operation?
Question: explain the repartition operation?
Question: explain the top and take ordered operation?
Question: explain the lookup operation?
Question: how to kill spark running application?
Question: follow me https www you tubecomcsauravagarwal49 how to stop info messages displaying on spark console?
Question: where the logs are available in spark on yarn?
Question: how to find out the different values in between two spark data frames?
Question: what are security options in apache spark?
Question: what is scala?
Question: what are the types of variable expressions available in scala?
Answer: val (aka Values): You can name results of expressions with the val keyword.Once refer a value, it does not re -compute it. Example: val x = 1 + 1 x = 3 // This does not compile. var (aka Variables): Variables are like values, except you can re -assign them. You can define a variable with the var keyword. Example: var x = 1 + 1 x = 3 // This can compile.
Question: what is the difference between method and functions in scala?
Question: what is case classes in scala?
Question: what is traits in scala?
Answer: Traits are used to share interfaces and fields between classes. They are similar to Java 8s interfaces. Classes and objects can extend traits  but traits cannot be instantiated and therefore have no parameters. Traits are types containing certain fields and methods. Multiple traits can be combined. A minimal trait is simply the keyword trait and an identifier: Example: trait Greeter {   def gr eet(name: String): Unit =  println("Hello, " + name + "!") }
Question: what is singleton object in scala?
Answer: An object is a class that has exactly one instance is called singleton object. Heres an example of a singleton object with a method: object Logger { def info(message: String): Unit = println("Hi i am Dineshkumar") }
Question: what is companion objects in scala?
Answer: An object with the same name as a class is called a companion object. Conversely, the class is the objects companion class. A companion class or object  can access the private members of its companion. Use a companion object for methods and values which are not specific to instances of the companion class. Example: import scala.math._ case class Circle(radius: Double) { import Circle._ Follow  Me : https://www.yo utube.com/c/SauravAgarwa l def area: Double = calculateArea(radius) } object Circle { private def calculateArea(radius: Double): Double = Pi * pow(radius, 2.0) } val circle1 = new Circle(5.0) circle1.area
Question: what are the special datatype available in scala?
Question: what is higher order functions in scala?
Question: what is currying function or multiple parameter lists in scala?
Answer: Methods may define  multiple parameter lists. When a method is called with a fewer number of parameter lists, then this will yield a function taking the missing parameter lists as its arguments. This is formally known as currying. Follow  Me : https://www.yo utube.com/c/SauravAgarwa l Example: def foldLeft[B](z: B)(op: (B, A) => B): B val numbers = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) val res = numbers.foldLeft(0)((m, n) => m + n) print(res) // 55
Question: what is pattern matching in scala?
Question: what are the basic properties avail in spark?
Question: what are the configuration properties in spark?
Answer: spark.executor.memory :- The maximum possible is managed by the YARN cluster whichcannot exceed the actual RAM available. spark.executor.cores: - Number of cores assigned per Executor which cannot be higher than the cores available in each worker. spark.executor.instances: - Numb er of executors to start. This property is acknowledged by the cluster if spark.dynamicAllocation.enabled is set to false. spark.memory.fraction: - The default is set to 60% of the requested memory per executor. spark.dynamicAllocation.enabled: - Overrides  the mechanism that Spark provides to dynamically adjust resources. Disabling it provides more control over the number of the Executors that can be started, which in turn impact the amount of storage available for the session. For more information, please see the Dynamic Resource Allocation page in the official Spark website.
Question: what is sealed classes?
Answer: Traits and classes can be marked sealed which means all subtypes must be declared in the same file. This is useful for pattern matching because we dont ne ed a catch all case. This assures that all subtypes are known. Example: sealed abstract class Furniture case class Couch() extends Furniture case class Chair() extends Furniture def findPlaceToSit(piece: Furniture): String = piece match { case a: Couch => "Lie on the couch" case b: Chair => "Sit on the chair" }
Question: what is type inference?
Answer: The Scala compiler can often infer the type of an expression so you dont have to declare it explicitly. Example: val Name = "Dineshkumar S" // it consider as S tring val id = 1234 // considered as int
Question: follow me https www you tubecomcsauravagarwal68 when not to rely on defaulttype inference?
Answer: The type inferred for obj was Null. Since the only value of that type is null, So it is impossible to assign a different value by default.
Question: how can we debug spark application locally?
Question: map collection has key and value then key should be mutable or immutable?
Answer: Behavior of a Map is not specified if value of an object is changed in a manner that affects equals comparison while object with the key. So Key should be an immutable.
Question: what is off heap persistence in spark?
Answer: One of the most important capabilities in S park is persisting (or caching) datasets in memory across operations. Each persisted RDD can be stored using a different storage level. One of the possibilities is to store RDDs in serialized format off -heap. Compared to storing data in the Spark JVM, off -heap storage reduces garbage collection overhead and allows executors to be smaller and to share a pool of memory. This makes it attractive in environments with large heaps or multiple concurrent applications. Follow  Me : https://www.yo utube.com/c/SauravAgarwa l
Question: what is the difference between apache spark and apache f link?