Lingk's fundamental design philosophy is related to big data architecture. Lingk tools make it easy to access big data capabilities. Here are some key concepts to understand:Data is always memoryData is cleared after processing is complete unless explicitly written.Data is always processed as a setThink in terms of managing groups of data not individual rows ( with FOR loops, for example).The full power of SQL is available to data setsFor conditional logic and filtering in Lingk Query Language (LQL), you can utilize the full capabilities of SQL. (see LQL statements)Data is always processed in the JSON format Nested structures and arrays have unique support in the LQL dialects due to the power of Spark SQLData can be automatically unioned from multiple API calls or files Creating sets of data from different data sources is made connectors that automatically join similar data sets together. (See ParameterizedBy)Create tables of data from other tables of data Similar to other SQL dialects, a powerful data processing technique is to create temp tables. In Lingk, these table are automatically dropped after execution. (see Recipe Execution Lifecycle)All data can be diffedData coming from any connector or format can be cached using the data diff post processor.  (see Data Diff)Null values require schema to be properly handledWhen dealing with null values, schema enables subsequent LQL operations referencing the field to find the column in the set and succeed.  (see Schema)