magento/magento2 — pull request #542 by tim-bezhashvyly (2014-04-06 22:12:37+00:00)
Title: Fix ImportExport bug which occurs while importing multiple rows per entity

This pull request solves couple of bugs.

As you know Magento allow import of a complex data splitting the entity information into multiple lines. So let's say I want to import 2 products like this:
sku,_store,_attribute_set,_type,_category,_root_category,_product_websites,activation_information,color,computer_manufacturers,contrast_ratio,cost,country_of_manufacture,country_orgin,cpu_speed,created_at,custom_design,custom_design_from,custom_design_to,custom_layout_update,description,dimension,finish,gallery,gender,gift_message_available,harddrive_speed,hardrive,has_options,image,image_label,in_depth,manufacturer,max_resolution,media_gallery,megapixels,memory,meta_description,meta_keyword,meta_title,minimal_price,model,msrp,msrp_display_actual_price_type,msrp_enabled,name,news_from_date,news_to_date,options_container,page_layout,price,processor,ram_size,required_options,response_time,room,screensize,shape,shirt_size,shoe_size,shoe_type,short_description,small_image,small_image_label,special_from_date,special_price,special_to_date,status,tax_class_id,thumbnail,thumbnail_label,updated_at,url_key,url_path,visibility,weight,qty,min_qty,use_config_min_qty,is_qty_decimal,backorders,use_config_backorders,min_sale_qty,use_config_min_sale_qty,max_sale_qty,use_config_max_sale_qty,is_in_stock,notify_stock_qty,use_config_notify_stock_qty,manage_stock,use_config_manage_stock,stock_status_changed_auto,use_config_qty_increments,qty_increments,use_config_enable_qty_inc,enable_qty_increments,is_decimal_divided,_links_related_sku,_links_related_position,_links_crosssell_sku,_links_crosssell_position,_links_upsell_sku,_links_upsell_position,_associated_sku,_associated_default_qty,_associated_position,_tier_price_website,_tier_price_customer_group,_tier_price_qty,_tier_price_price,_group_price_website,_group_price_customer_group,_group_price_price,_media_attribute_id,_media_image,_media_lable,_media_position,_media_is_disabled,_custom_option_store,_custom_option_type,_custom_option_title,_custom_option_is_required,_custom_option_price,_custom_option_sku,_custom_option_max_characters,_custom_option_sort_order,_custom_option_row_title,_custom_option_row_price,_custom_option_row_sku,_custom_option_row_sort,_super_products_sku,_super_attribute_code,_super_attribute_option,_super_attribute_price_corr
1111,,Furniture,simple,"Furniture/Living Room","Root Catalog",base,,Red,,,50.0000,,Italy,,"2007-08-28 16:25:46",,,,,"The Magento ottoman will impress with its style while it delivers on quality. This piece of living room furniture is built to last with durable solid wood framing, generous padding and plush stain-resistant microfiber upholstery.",,Microfiber,,,,,,0,/o/t/ottoman.jpg,,,,,,,,Ottoman,Ottoman,Ottoman,299.9900,magotto,,,,Ottoman,,,"Block after Info Column",,299.9900,,,0,,"Living Room",,,,,,"With durable solid wood framing, generous padding and plush stain-resistant microfiber upholstery.",/o/t/ottoman.jpg,,,,,1,2,/o/t/ottoman.jpg,,"2008-08-08 14:59:04",ottoman,ottoman.html,4,20.0000,706.0000,0.0000,1,0,0,1,1.0000,1,100.0000,1,1,,1,0,1,0,1,0.0000,1,0,0,1112,,384822,,1114,,,,,,,,,,,,703,/o/t/ottoman.jpg,,0,1,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,bar1234,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1112,,Furniture,simple,"Some Non Existent Category","Root Catalog",base,,Red,,,50.0000,,Italy,,"2007-08-28 16:29:24",,,,,"This Magento chair features a fun, futuristic design, with fluid curves and gentle angles that follow the shape of the body to enhance ultimate relaxation. It combines a hint of nostalgia with the up-to-date sensibility and function of modern chairs. It is in soft, velvety microfiber.",,Microfiber,,,,,,0,/c/h/chair.jpg,,,,,,,,Chair,Chair,Chair,129.9900,1112,,,,Chair,,,"Block after Info Column",,129.9900,,,0,,"Living Room",,,,,,"Combining a hint of nostalgia with the up-to-date sensibility and function of modern chairs. It is in soft, velvety microfiber.",/c/h/chair.jpg,,,,,1,2,/c/h/chair.jpg,,"2008-08-08 14:59:17",chair,chair.html,4,50.0000,724.0000,0.0000,1,0,0,1,1.0000,1,100.0000,1,1,,1,0,1,0,1,0.0000,1,0,0,1113,,384822,,1114,,,,,,,,,,,,703,/c/h/chair.jpg,,0,1,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,n2610,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

As you can see the only purpose of the second rows (let's call them "slave" rows) for each product is to provide one more cross-selling connection.

The first problem happens when importer will try to process the "master" row of the second product (SKU = 1112). The validation of the row will fail. In our case it will happen because of non-existing category but the reason can be different. The importer will rightfully skip this row and move to "slave" row. This row will pass validation but as the previous row was ignored it will be assigned to earlier product with SKU = 1111. This means the cross-selling connection will be added to wrong product.

This becomes even worse threat for multilingual stores where products can get wrong names, images or other vital attributes.

This pull request is addressing this issue by grouping rows which belong to the same entity and pushing them into the bunch only if all rows of the group passed validation. If one row of the group fails the validation the rest of the "slave" rows will not be even checked which will insignificantly speed up the processing.

If user prefers importer to process the group regardless if it contains invalid rows he can achieve it by adding `sku` values along with other required values to each "slave" row.

The second bug is somehow alike. It is derived from the fact that importer will split large data sets into chunks (by default of 500 rows) and save them for future processing. The problem is that "slave" rows which are wrapped into the next bunch are ignored and lost.

This pull request is addressing this issue by closing the bunch only if the there are no more "slave" rows left. So each bunch will always start with a "master" row.

Both bugs above are extremely critical and resulted in my complete catalog of 200k SKUs to be messed up, sales loss and significant damage to the business.