Repository: inventree/InvenTree

Issue:
Can't import all data fields when importing excel file of parts
**Describe the bug**
Dear All,
I'm using lastest InvenTree version on Ubuntu LTS 20.04 Server ( automatic dowload when install as described is doc https://inventree.readthedocs.io )
I've created parts manually and then export to excel file to get I/O excel files format, then I tried to upload all our stock ( of resistors ).
Only a few fields are availables in the graphical parser ( lack at least stock, stock min, price, location, etc ... )

A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Having a minimum filled database with parts and parts categories, and options for import excel set to on
2. Try to upload excel file with parts within a subcategory
3. Try to choose fields names
4. See error

**Expected behavior**
I expect to be able to import all the available fiels of database's dataset "part"

**Screenshots**
If applicable, add screenshots to help explain your problem ; I'll give them if needed, with pleasure.

**Deployment Method**
Bare Metal ( local, dev )

**Last but no least**
How to communicate with each others ? Only github ? no mailing list ?
How to contibute with testing ? Where to start ?
How to contribute with features ? Where to start ?
We are EMS so InvenTree can be our daily S/W and of course we are self-motivated to contribute in proposing features, maybe coding features, but where to start to "enter into the code" ?

Regards,
Lea & JCh.

Can't import all data fields when importing excel file of parts
**Describe the bug**
Dear All,
I'm using lastest InvenTree version on Ubuntu LTS 20.04 Server ( automatic dowload when install as described is doc https://inventree.readthedocs.io )
I've created parts manually and then export to excel file to get I/O excel files format, then I tried to upload all our stock ( of resistors ).
Only a few fields are availables in the graphical parser ( lack at least stock, stock min, price, location, etc ... )

A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Having a minimum filled database with parts and parts categories, and options for import excel set to on
2. Try to upload excel file with parts within a subcategory
3. Try to choose fields names
4. See error

**Expected behavior**
I expect to be able to import all the available fiels of database's dataset "part"

**Screenshots**
If applicable, add screenshots to help explain your problem ; I'll give them if needed, with pleasure.

**Deployment Method**
Bare Metal ( local, dev )

**Last but no least**
How to communicate with each others ? Only github ? no mailing list ?
How to contibute with testing ? Where to start ?
How to contribute with features ? Where to start ?
We are EMS so InvenTree can be our daily S/W and of course we are self-motivated to contribute in proposing features, maybe coding features, but where to start to "enter into the code" ?

Regards,
Lea & JCh.



Discussion:
@tse-tarn-france hi there! Thanks for fileing this. I will need to reporduce but from the top of my head I can say that not all fields are mapped to the importer.
So this is more of a feature request. Should not be to dificult.

> **Last but no least**
> How to communicate with each others ? Only github ? no mailing list ?

We communicate strictly via github, either in issues (either bug or feature request template)
> How to contibute with testing ? Where to start ?

We use the standard django testing suit - maybe have a look at the tests [here ](https://github.com/inventree/InvenTree/blob/50ae331afde1a752a8336e7e7bcaa8195503b45e/InvenTree/stock/tests.py) or the [general docs](https://docs.djangoproject.com/en/3.2/topics/testing/)
> How to contribute with features ? Where to start ?

We have a long list of issues/FR here in github. Simple issues are marked either with the `starter` or `help wanted` labels. Maybe search for something that intrests you and discucss where to start on the issueYou can mark the maintainer @SchrodingersGat or some key-contributers @eeintech, @matmair - we gladly help with specific problems. please remember this is a hobby for all involved. And Free in the meaning of `free beer` - no strings attached.
> We are EMS so InvenTree can be our daily S/W and of course we are self-motivated to contribute in proposing features, maybe coding features, but where to start to "enter into the code" ?

[Our docs](https://inventree.readthedocs.io/en/latest/contribute/) or [github](https://github.com/inventree/InvenTree/blob/master/CONTRIBUTING.md) point to some base rules - submitting PRs to [the docs](https://github.com/inventree/inventree-docs/) is also very welcome. Please search in the open issues and docs before you propose something.
@tse-tarn-france Bienvenue sur le projet InvenTree !

I can also try to reproduce but first, could you please confirm the steps are:
1. Create stock items using InvenTree UI
2. Use Admin interface to export newly created stock items to CSV/XLS sheet
3. Use Admin interface to re-import stock items from CSV/XLS sheet
4. Observe missing fields for import

Can you please also provide information:
1. InvenTree version information in this issue
2. Screenshot showing the missing import fields
3. Is the re-import done in the same instance or a different one? If different, are instances running with the same configuration?
4. Backend: SQLite, MySQL, PostGreSQL (might matter for the final import process)

Merci :wink: 
@tse-tarn-france I reproduced this and not all fields are exposed. I do not know how much of the documentation you have read but stock is not just a number. There are (one or multiple) stock items connected to the number so you can not just import a basic number.
Could you file which fields you are missing? The current fields are defined here
https://github.com/inventree/InvenTree/blob/22572c6f3505c5667e7b394959ca014427ed58f7/InvenTree/part/views.py#L234-L258
Dear all,

my Inventree 
![versions](https://user-images.githubusercontent.com/92796696/138143503-0ea68145-b14f-46ef-b0ee-4cbccca0dea0.jpg)
[InvenTree_Parts_Export.xls](https://github.com/inventree/InvenTree/files/7383227/InvenTree_Parts_Export.xls)
[InvenTree_Parts_Export.xls](https://github.com/inventree/InvenTree/files/7383231/InvenTree_Parts_Export.xls)
version is 0.6.0 dev

here are some screenshots + excel demo files. Indeed, I think being able to import all fields could be nice ( see export excel file ).

Enclosed my basic stock file to upload ( stock is "in_stock" ). minimum_stock is nice to import as well but is already possible.
import last buy price or expected buy price would be usefull, because when we buy, we need to know the last negociated price, and which is then expected. Moreover for annual inventory evaluation, last buy price is necessary.

Hope this help !
![1](https://user-images.githubusercontent.com/92796696/138143196-1a410412-cceb-44c0-be6f-f3f492780cfc.jpg)
![2](https://user-images.githubusercontent.com/92796696/138143205-5c93f494-facb-4386-872e-28e15aa44803.jpg)
![3](https://user-images.githubusercontent.com/92796696/138143208-971f42fd-a7d3-4868-abff-315866a5fd20.jpg)
![4](https://user-images.githubusercontent.com/92796696/138143215-4a65e6f1-53b5-488e-b237-a9c19ae3c795.jpg)
[STOCK_TSE_IN_062021_C3.csv](https://github.com/inventree/InvenTree/files/7383216/STOCK_TSE_IN_062021_C3.csv)

PS : I'm using sqlite
I did not realize you were using the part import, sorry for that @tse-tarn-france 
Importing part will not get you the stock items, you'll have to use the Admin interface in this case.

@matmair Maybe we need to be more clear what is and what is _not_ supported for the part import in the documentation?
Well, I tried to import from Admin page, but it does not do anything as well :

![admin_stock_import_upload](https://user-images.githubusercontent.com/92796696/138344006-0b86e4bf-fbf3-4520-9aef-b35fdc0c17e1.jpg)

And the result :+1:

![admin_stock_import_result](https://user-images.githubusercontent.com/92796696/138344074-d8717c14-68cb-437e-a4a4-14d6321697d4.jpg)

Maybe anyone has its import test file ( excel or csv ) ? as an exemple ?

Here is mine ( our current inventory for Condensators )

[STOCK_TSE_IN_062021_C3 - Copie.xls](https://github.com/inventree/InvenTree/files/7392377/STOCK_TSE_IN_062021_C3.-.Copie.xls)

Regards.

@tse-tarn-france In your file's header row, change:
* `REF` to `IPN`
* `Designation` to `name`

What do you get?

I get this on `stable` branch:
![image](https://user-images.githubusercontent.com/4020546/138356437-ad7ceb9b-7fc9-45d2-910d-8f2f73bb4319.png)

> 
> 
> I did not realize you were using the part import, sorry for that @tse-tarn-france Importing part will not get you the stock items, you'll have to use the Admin interface in this case.
> 
> @matmair Maybe we need to be more clear what is and what is _not_ supported for the part import in the documentation?

@eeintech I would rather add the missing fields as it is quite simple with the current wizard setup. I am just unsure what fields we really need. If stock is filled in we can create an initial stock item - just like with the part creation dialogue.
@matmair I think there is some confusion from OP, stock items cannot be created from part import. Also we need to hear from OP if the Admin import does or does not import those fields.
well I don't understand the point, because parts and stock are linked.

If you import parts definition from one side and then stock datas for parts from another side, you may get errors unless you link both imports with 1 field, which could be IPN or database primary key, but obviously the link element MUST be unique !

However import features are important for quick turn-on of InvenTree for existing compagnies, because existing compagnies already have parts and stock of parts. So imports MUST be easy for InvenTree to be widely used, which is the aim, no ? :-))

Regards.
@tse-tarn-france I get your point but `part` and `stock_item` are stored in complete separate tables. When you import `part` you cannot currently create `stock_item` just from the data in your spreadsheet. You need to first import the parts then the stock items. The stock item spreadsheet should have a link to each part it is linked to. That's how importing works in any database, step by step, and using relationship between tables.
> So imports MUST be easy for InvenTree to be widely used

If you want to work on making it easier, don't hesitate to contribute!
You have to understand that there is limited bandwidth from all the contributors and that your priorities are not everybody else priorities :wink: 
Users have imported data before so I'm sure you can do it too, just break it down into multiple spreasheet with each representing a database table and I'm sure you'll get there too!
You're right, users have imported data before so I'm trying to understand how they did ! :-)

Hope I'll be able to contribute with code mods soon !

And just to say it, import features is the only step I need to start to use InvenTree in our company. Others features could wait.

So last questions but not least :
 - how to regenerate the server after a change in the python code ?
 - how to switch between my current 0.6.0dev version and the 0.5.2 stable one ?
 - why when installing by using docs it choose 0.6.0dev which I can not find named like this in github ?

As you see, I'm a beginner :-)
@tse-tarn-france I will try to make a PR that includes the rest of the fields.



> So last questions but not least :
> 
>     * how to regenerate the server after a change in the python code ?
> 
If you are running with DEBUG enabled it auto-restarts, else you have to stop it and maybe run 'inv migrate' after you changed the models
>     * how to switch between my current 0.6.0dev version and the 0.5.2 stable one ?
> 
You have to install the version you want - there are git or docker tags. So if you want 0.5.2 you have to pull the version tagged 0.5.2 / we have a 'stable' channel for the latest stable version. 'master' is the current development version.
>     * why when installing by using docs it choose 0.6.0dev which I can not find named like this in github ?
If you do not specify the version and use master you get the current dev version.
@tse-tarn-france hi there! Thanks for fileing this. I will need to reporduce but from the top of my head I can say that not all fields are mapped to the importer.
So this is more of a feature request. Should not be to dificult.

> **Last but no least**
> How to communicate with each others ? Only github ? no mailing list ?

We communicate strictly via github, either in issues (either bug or feature request template)
> How to contibute with testing ? Where to start ?

We use the standard django testing suit - maybe have a look at the tests [here ](https://github.com/inventree/InvenTree/blob/50ae331afde1a752a8336e7e7bcaa8195503b45e/InvenTree/stock/tests.py) or the [general docs](https://docs.djangoproject.com/en/3.2/topics/testing/)
> How to contribute with features ? Where to start ?

We have a long list of issues/FR here in github. Simple issues are marked either with the `starter` or `help wanted` labels. Maybe search for something that intrests you and discucss where to start on the issueYou can mark the maintainer @SchrodingersGat or some key-contributers @eeintech, @matmair - we gladly help with specific problems. please remember this is a hobby for all involved. And Free in the meaning of `free beer` - no strings attached.
> We are EMS so InvenTree can be our daily S/W and of course we are self-motivated to contribute in proposing features, maybe coding features, but where to start to "enter into the code" ?

[Our docs](https://inventree.readthedocs.io/en/latest/contribute/) or [github](https://github.com/inventree/InvenTree/blob/master/CONTRIBUTING.md) point to some base rules - submitting PRs to [the docs](https://github.com/inventree/inventree-docs/) is also very welcome. Please search in the open issues and docs before you propose something.
@tse-tarn-france Bienvenue sur le projet InvenTree !

I can also try to reproduce but first, could you please confirm the steps are:
1. Create stock items using InvenTree UI
2. Use Admin interface to export newly created stock items to CSV/XLS sheet
3. Use Admin interface to re-import stock items from CSV/XLS sheet
4. Observe missing fields for import

Can you please also provide information:
1. InvenTree version information in this issue
2. Screenshot showing the missing import fields
3. Is the re-import done in the same instance or a different one? If different, are instances running with the same configuration?
4. Backend: SQLite, MySQL, PostGreSQL (might matter for the final import process)

Merci :wink: 
@tse-tarn-france I reproduced this and not all fields are exposed. I do not know how much of the documentation you have read but stock is not just a number. There are (one or multiple) stock items connected to the number so you can not just import a basic number.
Could you file which fields you are missing? The current fields are defined here
https://github.com/inventree/InvenTree/blob/22572c6f3505c5667e7b394959ca014427ed58f7/InvenTree/part/views.py#L234-L258
Dear all,

my Inventree 
![versions](https://user-images.githubusercontent.com/92796696/138143503-0ea68145-b14f-46ef-b0ee-4cbccca0dea0.jpg)
[InvenTree_Parts_Export.xls](https://github.com/inventree/InvenTree/files/7383227/InvenTree_Parts_Export.xls)
[InvenTree_Parts_Export.xls](https://github.com/inventree/InvenTree/files/7383231/InvenTree_Parts_Export.xls)
version is 0.6.0 dev

here are some screenshots + excel demo files. Indeed, I think being able to import all fields could be nice ( see export excel file ).

Enclosed my basic stock file to upload ( stock is "in_stock" ). minimum_stock is nice to import as well but is already possible.
import last buy price or expected buy price would be usefull, because when we buy, we need to know the last negociated price, and which is then expected. Moreover for annual inventory evaluation, last buy price is necessary.

Hope this help !
![1](https://user-images.githubusercontent.com/92796696/138143196-1a410412-cceb-44c0-be6f-f3f492780cfc.jpg)
![2](https://user-images.githubusercontent.com/92796696/138143205-5c93f494-facb-4386-872e-28e15aa44803.jpg)
![3](https://user-images.githubusercontent.com/92796696/138143208-971f42fd-a7d3-4868-abff-315866a5fd20.jpg)
![4](https://user-images.githubusercontent.com/92796696/138143215-4a65e6f1-53b5-488e-b237-a9c19ae3c795.jpg)
[STOCK_TSE_IN_062021_C3.csv](https://github.com/inventree/InvenTree/files/7383216/STOCK_TSE_IN_062021_C3.csv)

PS : I'm using sqlite
I did not realize you were using the part import, sorry for that @tse-tarn-france 
Importing part will not get you the stock items, you'll have to use the Admin interface in this case.

@matmair Maybe we need to be more clear what is and what is _not_ supported for the part import in the documentation?
Well, I tried to import from Admin page, but it does not do anything as well :

![admin_stock_import_upload](https://user-images.githubusercontent.com/92796696/138344006-0b86e4bf-fbf3-4520-9aef-b35fdc0c17e1.jpg)

And the result :+1:

![admin_stock_import_result](https://user-images.githubusercontent.com/92796696/138344074-d8717c14-68cb-437e-a4a4-14d6321697d4.jpg)

Maybe anyone has its import test file ( excel or csv ) ? as an exemple ?

Here is mine ( our current inventory for Condensators )

[STOCK_TSE_IN_062021_C3 - Copie.xls](https://github.com/inventree/InvenTree/files/7392377/STOCK_TSE_IN_062021_C3.-.Copie.xls)

Regards.

@tse-tarn-france In your file's header row, change:
* `REF` to `IPN`
* `Designation` to `name`

What do you get?

I get this on `stable` branch:
![image](https://user-images.githubusercontent.com/4020546/138356437-ad7ceb9b-7fc9-45d2-910d-8f2f73bb4319.png)

> 
> 
> I did not realize you were using the part import, sorry for that @tse-tarn-france Importing part will not get you the stock items, you'll have to use the Admin interface in this case.
> 
> @matmair Maybe we need to be more clear what is and what is _not_ supported for the part import in the documentation?

@eeintech I would rather add the missing fields as it is quite simple with the current wizard setup. I am just unsure what fields we really need. If stock is filled in we can create an initial stock item - just like with the part creation dialogue.
@matmair I think there is some confusion from OP, stock items cannot be created from part import. Also we need to hear from OP if the Admin import does or does not import those fields.
well I don't understand the point, because parts and stock are linked.

If you import parts definition from one side and then stock datas for parts from another side, you may get errors unless you link both imports with 1 field, which could be IPN or database primary key, but obviously the link element MUST be unique !

However import features are important for quick turn-on of InvenTree for existing compagnies, because existing compagnies already have parts and stock of parts. So imports MUST be easy for InvenTree to be widely used, which is the aim, no ? :-))

Regards.
@tse-tarn-france I get your point but `part` and `stock_item` are stored in complete separate tables. When you import `part` you cannot currently create `stock_item` just from the data in your spreadsheet. You need to first import the parts then the stock items. The stock item spreadsheet should have a link to each part it is linked to. That's how importing works in any database, step by step, and using relationship between tables.
> So imports MUST be easy for InvenTree to be widely used

If you want to work on making it easier, don't hesitate to contribute!
You have to understand that there is limited bandwidth from all the contributors and that your priorities are not everybody else priorities :wink: 
Users have imported data before so I'm sure you can do it too, just break it down into multiple spreasheet with each representing a database table and I'm sure you'll get there too!
You're right, users have imported data before so I'm trying to understand how they did ! :-)

Hope I'll be able to contribute with code mods soon !

And just to say it, import features is the only step I need to start to use InvenTree in our company. Others features could wait.

So last questions but not least :
 - how to regenerate the server after a change in the python code ?
 - how to switch between my current 0.6.0dev version and the 0.5.2 stable one ?
 - why when installing by using docs it choose 0.6.0dev which I can not find named like this in github ?

As you see, I'm a beginner :-)
@tse-tarn-france I will try to make a PR that includes the rest of the fields.



> So last questions but not least :
> 
>     * how to regenerate the server after a change in the python code ?
> 
If you are running with DEBUG enabled it auto-restarts, else you have to stop it and maybe run 'inv migrate' after you changed the models
>     * how to switch between my current 0.6.0dev version and the 0.5.2 stable one ?
> 
You have to install the version you want - there are git or docker tags. So if you want 0.5.2 you have to pull the version tagged 0.5.2 / we have a 'stable' channel for the latest stable version. 'master' is the current development version.
>     * why when installing by using docs it choose 0.6.0dev which I can not find named like this in github ?
If you do not specify the version and use master you get the current dev version.

Resolving patch:
diff --git a/InvenTree/part/views.py b/InvenTree/part/views.py
--- a/InvenTree/part/views.py
+++ b/InvenTree/part/views.py
@@ -42,11 +42,12 @@
 from common.views import FileManagementFormView, FileManagementAjaxView
 from common.forms import UploadFileForm, MatchFieldForm
 
-from stock.models import StockLocation
+from stock.models import StockItem, StockLocation
 
 import common.settings as inventree_settings
 
 from . import forms as part_forms
+from . import settings as part_settings
 from .bom import MakeBomTemplate, ExportBom, IsValidBOMFormat
 from order.models import PurchaseOrderLineItem
 
@@ -245,6 +246,7 @@ class PartFileManager(FileManager):
             'Category',
             'default_location',
             'default_supplier',
+            'variant_of',
         ]
 
         OPTIONAL_HEADERS = [
@@ -256,6 +258,17 @@ class PartFileManager(FileManager):
             'minimum_stock',
             'Units',
             'Notes',
+            'Active',
+            'base_cost',
+            'Multiple',
+            'Assembly',
+            'Component',
+            'is_template',
+            'Purchaseable',
+            'Salable',
+            'Trackable',
+            'Virtual',
+            'Stock',
         ]
 
     name = 'part'
@@ -284,6 +297,18 @@ class PartFileManager(FileManager):
         'category': 'category',
         'default_location': 'default_location',
         'default_supplier': 'default_supplier',
+        'variant_of': 'variant_of',
+        'active': 'active',
+        'base_cost': 'base_cost',
+        'multiple': 'multiple',
+        'assembly': 'assembly',
+        'component': 'component',
+        'is_template': 'is_template',
+        'purchaseable': 'purchaseable',
+        'salable': 'salable',
+        'trackable': 'trackable',
+        'virtual': 'virtual',
+        'stock': 'stock',
     }
     file_manager_class = PartFileManager
 
@@ -299,6 +324,8 @@ def get_field_selection(self):
         self.matches['default_location'] = ['name__contains']
         self.allowed_items['default_supplier'] = SupplierPart.objects.all()
         self.matches['default_supplier'] = ['SKU__contains']
+        self.allowed_items['variant_of'] = Part.objects.all()
+        self.matches['variant_of'] = ['name__contains']
 
         # setup
         self.file_manager.setup()
@@ -364,9 +391,29 @@ def done(self, form_list, **kwargs):
                 category=optional_matches['Category'],
                 default_location=optional_matches['default_location'],
                 default_supplier=optional_matches['default_supplier'],
+                variant_of=optional_matches['variant_of'],
+                active=str2bool(part_data.get('active', True)),
+                base_cost=part_data.get('base_cost', 0),
+                multiple=part_data.get('multiple', 1),
+                assembly=str2bool(part_data.get('assembly', part_settings.part_assembly_default())),
+                component=str2bool(part_data.get('component', part_settings.part_component_default())),
+                is_template=str2bool(part_data.get('is_template', part_settings.part_template_default())),
+                purchaseable=str2bool(part_data.get('purchaseable', part_settings.part_purchaseable_default())),
+                salable=str2bool(part_data.get('salable', part_settings.part_salable_default())),
+                trackable=str2bool(part_data.get('trackable', part_settings.part_trackable_default())),
+                virtual=str2bool(part_data.get('virtual', part_settings.part_virtual_default())),
             )
             try:
                 new_part.save()
+
+                # add stock item if set
+                if part_data.get('stock', None):
+                    stock = StockItem(
+                        part=new_part,
+                        location=new_part.default_location,
+                        quantity=int(part_data.get('stock', 1)),
+                    )
+                    stock.save()
                 import_done += 1
             except ValidationError as _e:
                 import_error.append(', '.join(set(_e.messages)))


Test patch: