if from sratch....we have few subscritpion   under which resource gr....we have block keyvalut logicapp....
new aks or old....new namespaces......how much cpu and memory cluster resource qouta....if we same or new request for new resource qouta ..same for monfo cluster or old to be dropeed....kafka....new topic...as producer consumer ....list of topics....where its consumed.....they need data in new format...
new db or old db

afg ocado 
similar design to afg food order content ifo ocado all the products
complete design ....not req for info for other partners....

take one story per epic per task....each indiviual....at the end....how many test cases we have created to testing team
procduct core....
import template....excel sheet...
repos name....under that we have to create the test case...

Gibraltor: In Nonprod configure sftp connection for logic app.

PIGW-222

raise tkt....pagerduty...webhook....
pager

already with abhisheskk....no fiedl for stock.....
data requirement ....
they need to 
8  month....relaeaseed six cuprus was process ing india was overridden...
dont impact existing partner
......
adding any partner...
abhishesk....compleete no arch approval


thet view existing....roll out for all partner as strategic....
partner and arch

no validtai quality
architect...beam.....

include architects
pechy naresh

connect with jiten
confirme for all partner to relaease rollout for all partner ....standadised way
revisit from solution perspective

I wanted to get confirmation with you regarding the today meeting to be done or not as lalitha has said that no addtional fields are required for stock and she has shared a finalized excel doc with me.
as per lalitha...only one thing is to be pending is abhishek wants approval from both architect pechya and ramesh (naresh) regarding their views on strategic stock migration design....so that roll out for all partner as strategic form....confirm for all partner to relaease rollout for all partner ....standadised way without affecting the ....dont impact existing partner...
like existing partner data is not overridden......



and can you suggest how shuld move foward like we have connect with Both archiects with beam team regarding approval of design for strategic stock enhacements

Thursday, April 3, 2025 3:00 PM - 4:30 PM
dyntrace recording 21:22 

https://hso67908.apps.dynatrace.com/ui/apps/dynatrace.classic.settings/ui/settings/builtin:logmonitoring.log-events?gtf=-2h&gf=all&filter=summary%3Afilenot&id=71786c94-4f37-34c8-895b-962ae4a96937
https://hso67908.apps.dynatrace.com/ui/apps/dynatrace.classic.settings/#settings/syntheticprivatelocations;gtf=-2h;gf=all
https://docs.dynatrace.com/docs/observe/digital-experience/synthetic-monitoring
https://hso67908.apps.dynatrace.com/ui/apps/dynatrace.synthetic/monitors
https://hso67908.apps.dynatrace.com/ui/apps/dynatrace.launcher/getting-started
https://kqi08127.apps.dynatrace.com/ui/
 
 https://hso67908.apps.dynatrace.com/ui/apps/dynatrace.classic.problems/ui/problems?gtf=-2h&gf=3601333513198647657
 https://hso67908.apps.dynatrace.com/ui/apps/dynatrace.classic.settings/ui/settings/builtin:logmonitoring.log-events?gtf=-2h&gf=all&filter=summary%3Afilenot&id=71786c94-4f37-34c8-895b-962ae4a96937
 
 https://jira-marksandspencer-app.atlassian.net/wiki/spaces/II/pages/158010331/Dynatrace+Set+Up
 
https://jira-marksandspencer-app.atlassian.net/wiki/spaces/II/pages/158009633/Partner+Stock+Movement+Integration+Strategic+Approach

https://github.com/DigitalInnovation/International-PIGW-PartnerSalesIntegration/tree/development/partner-sales-service

firstcustom exception
Summary
International_PIGW_Prod_A2047_Alert_Admin_InvalidJobFrequencyException
International_PIGW_Prod_A2047_Alert_Admin_InvalidJobFrequencyException 

matcher
matchesValue(k8s.deployment.name,"*itl-pigw-admin-api-prod*") 
and matchesPhrase(content,"Job Frequency is invalid")

Title : Job Frequency is invalid

event type : custom alert (select)

Allow merge : yes

Set of additional key-value properties
PagerdutyServiceName: PIGWProd
Team: International-PIGW


Second custom exception

Summary
International_PIGW_Prod_A2047_Alert_ProductExtractor_CloudFileUploadException 

matcher
matchesValue(k8s.deployment.name,"*itl-product-extractor-prod*") 
and matchesPhrase(content,"CLOUD FILE UPLOAD FAILURE")

Title : CLOUD FILE UPLOAD EXCEPTION

event type : custom alert (select)

Allow merge : yes

Set of additional key-value properties
PagerdutyServiceName: PIGWProd
Team: International-PIGW


Summary

International_PIGW_Prod_A2047_Alert_ProductExtractor_FileNotFoundException

matcher


matchesValue(k8s.deployment.name,"*itl-product-extractor-prod*")
and matchesPhrase(content,"FILE NOT FOUND EXCEPTION")
Title

FILE NOT FOUND EXCEPTION

Event type

custom alert (select)

Allow merge

yes

Set of additional key-value properties

PagerdutyServiceName: PIGWProd
Team: International-PIGW


query=

schema mapping provide them ...with abhishesk ...will have a chat with them

we had asked for new field mappings but discussion with beam...was pending...
is there any existing parameters....that we need to enhance........
what was said 
add the paarameteers....add in summarise part....instead of that....include fields in 
schema mapping ....

beam...we dont need ....existing functionality....somefield we are sending discountprice custmerdetails...
new add....field....dto....
coomon field for validation....seeperate method for seperate partners....
but this has common validator....
plan for what....on stock...implemetation for solving what problem....common part...
partner sales services....
chcech india partner specfic fields...as part of the genreic...
schema is commonn....we can have ....
we need to india partner......we can discuss with beam and partner

do we need partner discussion also ....along with beam


https://jira-marksandspencer-app.atlassian.net/browse/PIGW-265
https://jira-marksandspencer-app.atlassian.net/browse/PIGW-266


 aks i hope
no azure function
blob to file format variability....
whcih 
specific file formats do we expect from our partners (e.g., CSV, XML
expected output is json
What transformation rules will be applied to convert various formats into the desired JSON structure?
controller way 


mapping table
mapping with file path
create execel of field ......reading of file and its output....
input file format and type and how many partners
processing of that fields ...whwhat is type of types field...valid for beam....
output file...how we are sneding...whcih logic...which inteface...


mustakim
whta are file we receieving and file we r generating
field we are receving...
stock source file anme....what is the source file name...for india csv...what destination and type of file generating for india...look in coding

which  r there ....which r the thing we need to generate.....


beam...we dont need ....existing functionality....somefield we are sending discountprice custmerdetails...
new add....field....dto....
coomon field for validation....seeperate method for seperate partners....
but this has common validator....
plan for what....on stock...implemetation for solving what problem....common part...
partner sales services....
chcech india partner specfic fields...as part of the genreic...
schema is commonn....we can have ....
we need to india partner......we can discuss with beam and partner

lalitha
undertsand how to process the file in blob which partner will put with file formt variability.....so that final is most probably a json to be send to beam...
after we have this doc at high level.....we will present to engineering team to discuss low level things...whether this present strategic design holds good for servicing
all partners as we had issues once during prod rolllover for one of the partner


need to discuss it beam as well and confirm if I mean from the data contract
there are some additional field also need to Into them
structure and everything need to be discussed 
Data mapping and data contract also see we have to discuss with beam
to analyze and understand if something we need from partner then we will start discussing with partner as well
and prepare a data contract
whatever we are asking partner should be able to provide and the other fields they are OK to share it across
what exactly expectation of beam, right?
What kind of what?
What is the data they require?
In documentation, strategic stock finance schema with validation rules

beam discuss with beam and prepare some data contract

Because in a strategic feed how we will send a stock data to them
So because right now, whatever is sending that is not enough
So there are some additional field also need to Into them and then.
Apart from this, like a structure and everything need to be discussed with them as well, right?


But here what we are talking about, I think we should get a discuss with beam as well. Right? And then?
Data mapping and data contract also see we have to discuss with beam, right and then maybe at the same time then we have to analyze and understand if something we need from partner then we will start discussing with partner as well.
I think there was a discussion as well. OK, so maybe I think that.

so design and then discussion with beam and prepare a data contract.


And then from partner perspective for stock?
I mean, we need to have some clarity around that. I think whatever we are asking partner should be able to provide, but they are also taking at their end how they can bring the intransit information as they was updating earlier. So that alone might be like, I mean we.
Need to discuss with partners, but other than that the other fields they are OK to share it across.
But we need to discuss it beam as well and confirm if I mean from the data contract SLS from the design perspective.


jiten=
So let's see.
Let's first try to get it closed with beam that is internal one, because then only we do understand, right?
So what exactly expectation of beam, right?
What kind of what?
What is the data they require?
OK. And then based on that, you know we can prepare and then discuss with partner.


In documentation, strategic stock finance schema with validation rules.
KA

JIten had told in last review call to have first try to get our understanding closed with beam team that is internal one, because then only we have finalization on
what exactly expectation of beam
What structure of data they need?
What is the data they require?
And then based on that, we can prepare our stock field mapping structure and then discuss with partner forward




please confirm if today’s meeting is needed, as Lalitha mentioned no additional stock fields are required and its already been discussed with beam team and shared a finalized Excel file with me.
As per Lalitha.....The only pending item is Abhishek’s request for approval from both architects Pechya and Ramesh (or Naresh) on the strategic stock migration AKS based design to ensure a standardized rollout for all partners without impacting existing data like it had happened for India during check partner  rollout

Let's discuss once with Lalitha

I am not aware about this

you had told me in last review call to have first try to get our understanding closed with beam team that is internal one, because then only we have finalization on

what exactly expectation of beam

What structure of data they need?

What is the data they require?

And then based on that, we can prepare our stock field mapping structure and then discuss with partner forward
 








---------------------------------------




trigger azure function.....file upload in blob or http request....saving in table or queue
get the file ....do some converson.....send to upstream
set of partner posting data chech india  finland.....setup storage sftp acc ....we do json conversoion....for each partner we dont have to do separate conversion logic...
we will have created the common schema ....we will map the field from the files from partner accooding  to the common schema....after that we will post this json ...
to the beam share path... as per the common schema during json conversion......

high level ....all partner will be posting the file to our SFTT storage account....we created user conn string...india checch....whitelist their ips....
post file to our container by each partner as per conn string seperately......
tactical....directly .....we are doing any conversion....directly we get file from partner in this blob folder ....logic apps will copy file from blob folder to our storage acc....
from there beam will come and consume it...

strategic approach....once file comes to this storage container account...we will download the file...do json conversion...we post file to beam share path...beam directly can
use for reporting purpose 

we are planning for chech sales for strategic approach....zalanado  further we can plan to shift to strategic approach

for chech partner partner code is 8468 so to this particular directory 8468 inbound in this folder container blob storage acc sftp this chech partner will be posting the file.
there chech azure function app....logic apps will copy file from blob folder to our storage acc....
from there beam will come and consume it...So what logic app will do is to this container, it will copy the file whatever file we got from partner in that SFTP first it will copy to this inbound folder container which with that particular file content....
So we don't have to do duplicate things.................So every day once after processing we have to delete the file also from this inbound.
So to do it as a reference we have created a file storage account there we have created as an archive folder.

beam team ...introducing 2 new feields....implemet few....store sales schema....we have to verofy....whcih interface....


-----------------------------------------------------

https://www.umletino.com/umletino.html


 springboot java azure hosted environemnet
 partnercode/inbound/interfacename
 
 
 8468/inbound/i3555.....
 chech partner will post the file here sftp everyday
 partnersalesnonprod logic app
 it will copy the file ...inbound container....002 from 001
 create archive folder and copy file
 and delete the file as we are daily getting 
 
 
 once file come to 002 container/inboiund folder....inbound/path/filename.... they azure function chech sales will blob trigger automatically...chech sales azure function...
 conversion to json schema and to outbound folder json file
 
 get configuration from table to get partnercode
 
 read the file....normally....we get txt file fromm partner....
 change to txt....if partner sending non txt file
 so that other extractor code we dont have to change
 
 common json schema.....map fields in txt file based on on this schema only to produce final json to be sent to beam 
 there final json strcuture will be based on comoon json schema file
 So according to the schema we have to form the Jason and then finally the Jason structure, we will form it.
 
header - in that schema if you see we have something 
 We have a header, in header we have message ID, create a time stamp, source application....interfacename
 
So these are some these are all static values in header we can actually get it from the database itself.
 
 
payload type - dynamic section values which we have to get from the file what the partner txt file is sending.
 
So from the partner txt file we will get the values from the columns like site ID, site name, everything we will get it and then we will actually map it to the final Jason structure.
Suppose if the partner is not sharing this GP in UPC and all right.
So this field itself we we will not be sending to beam in the final Jason structure.

So if we are having values for some of the fields, then only we will map it.

So these are the values we will have in said thrice we have these values like that this structure will be common for all the partners
Only thing is the values will be different and for some partner they will be sharing some more fields
So for that field alone will be coming over here.
So it will be a common structure which we will be following for all the partners

processor.process method calling

upload  file from storage acc container 002 inbound  to storage container  002 /outbound/path/filename upload (like /outbound/8468) the file to our storage acc....java process
 move file from our storage acc container outbound our storage acc....to /landing/inbound/intl_partner_sales/ beam share path storage acc to share file with beam....second logic app process - BeamConnectionNOnprod logic app
 trigger after file is upoad to blob /outbound/8468 - second logic app process BeamConnectionNOnprod logic app
 copy to archive also....by 2nd logic app
 we use second logic app file as we dont have any trigger for file storage transfer upload in outbound...we only have blob upload trigger...
 we will do blob container transfer 002 outbound/path/filename upload.....and not file share tranfer
 we will do file share path tranfer also for archive 
 we will delete from inboiund and outbound containers also
 we will archive also .... 
 

What this audit batch table will do is like how we have done for sales talk and all.
It will just track it actually then the process has started, what is the status of the upload.
So those things they will track using this audit batch table and it will update the status also in the table.
And finally we will close this connection.


sql queries 
There is a separate file for queries to get the data from the table like partner code, partner name and all right.


 

 -----------------------------
 yeah you mentioned like in file share only 50 MB file we could able to upload right
 
 Workaround: To work around this limitation, developers may use other methods, such as polling (periodically checking the file share for changes) or using an intermediate service that can trigger workflows based on file changes in the file share.
 
 BLOB Container Triggers: Allow for immediate reactions to changes, suitable for handling large files and unstructured data.
File Share Triggers: Generally do not exist, requiring alternative methods to detect and respond to file changes.

Data Structure: File shares have a hierarchical structure, while BLOB containers have a flat structure.
Access Protocol: File shares use SMB, making them suitable for direct file access; BLOB storage uses HTTP/HTTPS, ideal for web-based applications.
Collaboration: File shares are better for collaborative file access, whereas BLOB storage excels in handling unstructured data at scale.
Automation: BLOB containers support event-driven automation through triggers, while file shares require alternative methods for automation.


 
 
 -----------------------
 XXXXXXXXXXXXXXXXXXXXXXxXXXXXXXXXXXXXendXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 ---------------------------------------------------------------------------------------






























-----------------------------------------------------------------------

To create a FileNotFoundException in a Spring Boot application that can be captured in Dynatrace's problem notification rules, you will generally follow these steps:

Throwing the Exception: You need to write some code where you simulate or throw a FileNotFoundException. This might happen during file access or reading operations.
Handling the Exception: You can choose to handle it in a way that makes it obvious in logs or through reporting mechanisms that can be observed by Dynatrace.
Sending the Notification to Dynatrace: For Dynatrace to capture this exception and trigger notifications, you need to ensure that it is integrated properly with your Spring Boot application.
Step 1: Throwing the Exception
Here's a simple example of how you might throw a FileNotFoundException in a Spring Boot controller or service:


import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;

@RestController
public class FileController {

    @GetMapping("/read-file")
    public String readFile() throws FileNotFoundException {
        // Change the path to a file that doesn't exist on purpose
        File file = new File("nonexistentfile.txt");
        Scanner scanner = new Scanner(file);
        return "File content: " + scanner.nextLine();
    }
}
Step 2: Handling the Exception
You can create a global exception handler using @ControllerAdvice to manage how you want the exception to be reported:


import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.ExceptionHandler;

@ControllerAdvice
public class GlobalExceptionHandler {

    @ExceptionHandler(FileNotFoundException.class)
    public ResponseEntity<String> handleFileNotFoundException(FileNotFoundException ex) {
        // Here, you can log the exception, send it to Dynatrace, etc.
        System.err.println("File not found: " + ex.getMessage()); // This will be logged

        // Custom response can be returned
        return new ResponseEntity<>("File not found: " + ex.getMessage(), HttpStatus.NOT_FOUND);
    }
}
Step 3: Capture in Dynatrace
Ensure Dynatrace is configured: Make sure that Dynatrace agent is installed and correctly configured in your Spring Boot application. You might want to ensure that the agent is able to capture unhandled exceptions and logs.
Log the Exception: Ensure that your application logs the exception properly. Dynatrace can capture logs and exceptions if the logs are being collected by it.
Problem Notification Rules in Dynatrace: In Dynatrace, navigate to the problem notification settings and create rules to alert on exceptions. You can set criteria to match FileNotFoundException based on your logging strategy.
Example Log Statement
You can use SLF4J or any logging framework to log the exception message, making it easier for Dynatrace to capture:


import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@ControllerAdvice
public class GlobalExceptionHandler {

    private static final Logger logger = LoggerFactory.getLogger(GlobalExceptionHandler.class);

    @ExceptionHandler(FileNotFoundException.class)
    public ResponseEntity<String> handleFileNotFoundException(FileNotFoundException ex) {
        logger.error("File not found: {}", ex.getMessage());
        return new ResponseEntity<>("File not found: " + ex.getMessage(), HttpStatus.NOT_FOUND);
    }
}
Summary
By throwing a FileNotFoundException, handling it globally, logging it, and ensuring that Dynatrace is correctly capturing logs and exceptions, you can effectively create a scenario that meets your requirements. From this point, you'll need to verify that your Dynatrace setup is correctly monitoring and sending notifications based on the defined rules.


lalitha....beam...metting.....confirm...kya chahiye.....
meeting saath mein baith k ye field validation list....
iss product.....ye validation laga hai

all partner files are send to beam...sometimes we need to reupload maually and then it logic app triggered automatically

==============================================


Kumar, Alok started transcription

Natarajan, Lalithambal
0 minutes 4 seconds0:04
Natarajan, Lalithambal 0 minutes 4 seconds
At the end.
Natarajan, Lalithambal 0 minutes 9 seconds
The same design holds good or holds as a strategy question of the stock integration.
Natarajan, Lalithambal 0 minutes 15 seconds
This is something, and it may roll out for all the partners.
Natarajan, Lalithambal 0 minutes 18 seconds
Is that I mean that is a discussion that we need to have with the architects to be within.
Natarajan, Lalithambal 0 minutes 25 seconds
So I need to have a call scheduled along with data engineering team. Along with that we can align the current design that we have something that this.
Natarajan, Lalithambal 0 minutes 36 seconds
Can we rolled out for all the strategy partners as well?
KA

Kumar, Alok
0 minutes 45 seconds0:45
Kumar, Alok 0 minutes 45 seconds
Linsa what is the expectation in this?


Natarajan, Lalithambal
0 minutes 46 seconds0:46
Natarajan, Lalithambal 0 minutes 46 seconds
I mean requirements, ways.
Natarajan, Lalithambal 0 minutes 50 seconds
Declared.
Natarajan, Lalithambal 0 minutes 53 seconds
The same requirement, but whatever that we are developing currently that should something be used for all the international partners. So we need architects alignment to confirm whether this is the version that we can go ahead with strategic mean, go ahead as strategic because earlier when we had a.
Natarajan, Lalithambal 1 minute 12 seconds
Discussion. I mean we were facing one of the issues when we are deploying Cyprus after India.
Natarajan, Lalithambal 1 minute 20 seconds
So in the stockings for the deploying we had some issues in there.
Natarajan, Lalithambal 1 minute 25 seconds
So data engineering team wanted to confirm if you go into the same design, whether it have any issues subsequently when when we roll out the other partners or the same same design holds good for rolling out for the strategic, I mean rolling out as a strategy question for.
Natarajan, Lalithambal 1 minute 44 seconds
All the partners.
Natarajan, Lalithambal 1 minute 47 seconds
To get that agreement, we need to have a discussion and set up to agree on the design and based on the final design, agreed, we'll proceed on the work for this talk in.


Kumar, Alok
2 minutes2:00
Kumar, Alok 2 minutes
OK, I did not.
Kumar, Alok 2 minutes 2 seconds
Thanks for the info.
Kumar, Alok 2 minutes 5 seconds
Yeah. Thank.
Kumar, Alok 2 minutes 6 seconds
I'm good, thanks.

As you know, Latha was strategic sales, right, so.
Yadav, Jitendra 4 minutes 27 seconds
Stock sales, OK.
Yadav, Jitendra 4 minutes 28 seconds
So so not a stock integration, OK with the data integration?
Yadav, Jitendra 4 minutes 32 seconds
So yeah, could you please just, you know prepare some sample, OK.
Yadav, Jitendra 4 minutes 36 seconds
So basically, whenever we do a call with them, right?
Yadav, Jitendra 4 minutes 38 seconds
So basically we can use the same.
Yadav, Jitendra 4 minutes 42 seconds
How actually we are going to assert OK.
Yadav, Jitendra 4 minutes 45 seconds
So what is our expectation?
Yadav, Jitendra 4 minutes 47 seconds
How files to comes to us once file basically will come to us then how exactly we will process OK so solution at a high level, OK.
Yadav, Jitendra 4 minutes 55 seconds
So just put some boxes.
Yadav, Jitendra 4 minutes 58 seconds
And yeah, and then basically we can, OK.
KA
Kumar, Alok
5 minutes 3 seconds5:03
Kumar, Alok 5 minutes 3 seconds
Oh yeah, definitely.
Kumar, Alok 5 minutes 5 seconds
I've just started and trying to understand first what is the expectation that we exactly what we have to achieve, what is the requirement and we are looking. So after I understand myself then I will just definitely put all the boxes.

Yadav, Jitendra
5 minutes 11 seconds5:11


Yadav, Jitendra
5 minutes 11 seconds5:11
Yadav, Jitendra 5 minutes 11 seconds
Yeah, but a high level. Yeah. OK, fine. Correct. Correct.
KA
Kumar, Alok
5 minutes 18 seconds5:18
Kumar, Alok 5 minutes 18 seconds
Those are, yeah.

Yadav, Jitendra
5 minutes 18 seconds5:18
Yadav, Jitendra 5 minutes 18 seconds
No, no.
Yadav, Jitendra 5 minutes 19 seconds
See. Basically, no, no.
Yadav, Jitendra 5 minutes 20 seconds
See technical side OK.
Yadav, Jitendra 5 minutes 22 seconds
So basically technical side, you'll will get a file OK OK then basically file we have to process and then we have to generate data for them. OK so file format will be different. OK, so different partner may send in a different format. OK, so we'll get the file.
Yadav, Jitendra 5 minutes 39 seconds
Then basically we have to transform that file in a common format and common format is basically expected Jason for being OK, so high level it should be like this.
Yadav, Jitendra 5 minutes 48 seconds
So mostly file, but they'll drop in, you know BLOB. OK so basically this.
Yadav, Jitendra 5 minutes 53 seconds
At a high level, this will be the same thing. This thing and low level things.
Yadav, Jitendra 5 minutes 56 seconds
Basically, what data they need. OK, so low level things we will discuss with them, OK.
KA
Kumar, Alok
6 minutes 3 seconds6:03
Kumar, Alok 6 minutes 3 seconds
OK, OK.

Yadav, Jitendra
6 minutes 5 seconds6:05
Yadav, Jitendra 6 minutes 5 seconds
OK.
Yadav, Jitendra 6 minutes 5 seconds
Thank you.
KA
Kumar, Alok
6 minutes 6 seconds6:06
Kumar, Alok 6 minutes 6 seconds
That information was.
Kumar, Alok 6 minutes 10 seconds
Thanks for.

Natarajan, Lalithambal
6 minutes 12 seconds6:12
Natarajan, Lalithambal 6 minutes 12 seconds
And look, once you have those ready, kindly confirm you can plug in discussion with me as soon as.
KA
Kumar, Alok
6 minutes 19 seconds6:19
Kumar, Alok 6 minutes 19 seconds
Yeah, yeah.

Ghosh, Apurba
6 minutes 23 seconds6:23
Ghosh, Apurba 6 minutes 23 seconds
Atindra.


------------

The discussion highlights several important points regarding the integration and processing of data from various partners. Here are the key takeaways:

Design Strategy Alignment: Lalithambal emphasized the need for alignment with the architects and data engineering team to ensure that the current design can accommodate all strategic partners. This is crucial to avoid issues encountered during previous rollouts.

Clarification of Requirements: Alok raised the importance of understanding the expectations and requirements before proceeding. This aligns with Jitendra's request for a high-level overview of how files will be processed, indicating that clarity on requirements is essential for effective project execution.

Sample Preparation for Calls: Jitendra suggested preparing a sample solution to illustrate the process of file handling and transformation, which will help facilitate discussions with partners.

File Format Variability: Jitendra pointed out that different partners may send files in various formats, necessitating a common format (e.g., JSON) for processing. This highlights the technical complexity involved in standardizing data inputs.

Next Steps: It seems there is a consensus on scheduling a follow-up discussion once the necessary samples and clarifications are prepared. This collaborative approach will ensure that all stakeholders are on the same page moving forward.

---------------------------------

Step 4: Standardize Data Formats
Determine Common Format: Agree on a standard file format (e.g., JSON) for all partners to use.
Create Guidelines: Develop guidelines for partners on how to format their data submissions.

Step 5: Develop Sample Solutions
Prepare Sample Files: Create sample data files that illustrate the required format and processing steps.
Conduct Internal Testing: Test these samples within your team to ensure they meet the requirements.



undertsand how to process the file in blob which partner will put with file formt variability.....so that final is most probably a json to be send to beam...
after we have this doc at high level.....we will present to engineering team to discuss low level things...whether this present strategic design holds good for servicing
all partners as we had issues once during prod rolllover for one of the partner

------------------------------------------


https://github.com/DigitalInnovation/International-PIGW-PartnerSalesIntegration/blob/66e0e245a55f85f266b055f9c1952191bdc84df9/partner-sales-service/src/main/java/com/mns/intl/pigw/partner/sales/validation/price/CommonFieldValidator.java

In my local,The springboapplication application failed to start in my local because it could not find a required bean of type CommonFieldValidator for the PriceItemCyprusValidator class constructor.
Could you please take a look regarding the expected behavior?
I bilieve.....you need to annotate this class with @Component for it to be detected as a bean by the application context.
need to confirm 

-------===========================

The transcription covers a technical discussion between two individuals, Alok Kumar and Mustakim Patel, regarding an implementation of a system that handles stock, pricing, and partners. Here is a summary of the key points discussed:

Overview of the Task: They discuss the different components required for processing stock data, including understanding various files and their formats, the database structure, and partner interfaces.
Input and Output Files: Emphasis is placed on the importance of knowing the input file types and how they will be processed to generate output files. They talk about the need to check the source files, their naming patterns, and the specific data fields that must be read and validated.
Field Mapping and Validation: They mention creating a mapping table and an Excel sheet that maps input fields to output fields. They also highlight the importance of identifying mandatory and optional fields in the input files and how those will be validated during processing.
Modularity and Code Structure: The concept of creating separate modules for different functionalities is discussed. They propose splitting responsibilities among modules to improve maintainability and reduce code complexity. This discussion leads to considerations about microservices architecture and how to handle processing for multiple partners independently without affecting each other.
Challenges in Current Implementation: They identify challenges such as the size of the codebase currently in use and the limitations of existing functions that may impact performance. They also mention the potential need for thread management when processing large files in parallel.
Discussion about Azure Functions: The conversation includes mentions of using Azure Functions to execute various tasks and the limitations associated with them, including version constraints.
Next Steps: They conclude with the need to gather more information from team members like Lalitha and Jitendra on specific details such as field mappings and code structure before moving forward with implementation.
Overall, the session looks collaborative, with both individuals working together to figure out the best approach for a structured and efficient data handling system.

============================================

The transcript appears to be a conversation between two individuals, Alok Kumar and Mustakim Patel, discussing various aspects of a project related to stock management, data processing, and system interfaces. Here’s a detailed breakdown of the key points discussed:

Data Sources and Processing:

They talk about different components involved in their project, including the selling price, stock details, and the need to gather data from various sources.
There is mention of a file configuration which includes partners and input/output file paths.
Mapping Schemas:

The conversation touches on the importance of mapping schemas and field mappings for data processing. They discuss how fields will be read from input files and how they should be structured in output files.
Alok is tasked with creating an Excel sheet to document these mappings and ensure clarity on how fields are processed.
Validation Logic:

Mustakim emphasizes the need for validation checks, particularly distinguishing between mandatory and optional fields. They discuss how to handle invalid data and the importance of ensuring correct data types (e.g., numeric vs. alphanumeric).
Implementation Strategy:

They discuss structuring the code and the need for separate modules for different functionalities (stock, selling price, etc.) to enhance maintainability and reduce complexity.
The concept of microservices is introduced, suggesting that separating functionalities into distinct services can improve scalability and manageability.
Potential Challenges:

Alok and Mustakim highlight challenges regarding file processing, particularly when dealing with large files and multiple partners. They discuss how simultaneous processing can lead to issues with logging and error handling.
Future Considerations:

There is a consideration of using Azure Functions for processing but also an acknowledgment of its limitations, particularly regarding Java version upgrades.
They plan to communicate their findings and proposals to a third party (Lalitha) for further guidance and validation.
Project Coordination:

The need for clear communication and documentation is emphasized to ensure all team members are aligned on the project requirements and implementation strategies.
Overall, the conversation reflects a collaborative effort to define the technical requirements, challenges, and implementation strategies for a data processing project involving stock management and partner interactions.

--------------------------------------------------------------
Rephrased Summary
Alok Kumar and Mustakim Patel engaged in a discussion about their project related to stock management, focusing on data processing, validation, and system interfaces. They covered several key aspects including data sources, mapping schemas, validation logic, implementation strategies, challenges faced, and future considerations.

Data Sources and Processing: They discussed the components necessary for gathering stock and selling price data from various sources and the importance of file configurations that include partner details and input/output paths.

Mapping Schemas: The conversation highlighted the significance of mapping schemas for data processing. Alok was assigned to create an Excel sheet documenting field mappings for clarity on how data is structured in both input and output files.

Validation Logic: Mustakim emphasized the need for robust validation checks to differentiate between mandatory and optional fields, addressing how to manage invalid data types effectively.

Implementation Strategy: They talked about structuring the code with separate modules for different functionalities (such as stock and selling price) to enhance maintainability and reduce complexity. The concept of microservices was introduced to improve scalability.

Potential Challenges: Challenges regarding large file processing and simultaneous operations were discussed, particularly concerning logging and error handling.

Future Considerations: They considered using Azure Functions for processing but noted its limitations, particularly related to Java version upgrades. They planned to discuss their findings with Lalitha for further guidance.

Project Coordination: Clear communication and documentation were emphasized to ensure all team members remain aligned with the project requirements and implementation plans.

Action Items
Create Excel Mapping Sheet:

Who: Alok Kumar
What: Document the mapping of fields from input files to output files.
Deadline: [Specify deadline]
Define Validation Logic:

Who: Alok Kumar
What: Establish criteria for mandatory and optional fields and outline validation checks for data types.
Deadline: [Specify deadline]
Develop Separate Modules:

Who: Mustakim Patel and Alok Kumar
What: Design and implement separate modules for stock, selling price, and other functionalities.
Deadline: [Specify deadline]
Assess File Processing Challenges:

Who: Mustakim Patel
What: Identify potential issues with large file handling and address logging and error management strategies.
Deadline: [Specify deadline]
Discuss with Lalitha:

Who: Alok Kumar and Mustakim Patel
What: Present findings and proposals regarding implementation strategies and gather feedback.
Deadline: [Specify deadline]
Evaluate Azure Functions:

Who: Alok Kumar
What: Research the limitations of Azure Functions and assess alternatives for future upgrades.
Deadline: [Specify deadline]
Ensure Project Documentation:

Who: Alok Kumar and Mustakim Patel
What: Maintain clear documentation of project requirements and implementation plans for team alignment.
Deadline: Ongoing
These action items will help structure the project moving forward and ensure clarity and accountability among team members.



-----------------------------------------------------------

In the provided code for PartnerSalesProcessor, the stock-related processing can be divided into several major sections. Each section focuses on a different aspect of handling stock data. Here’s a breakdown of the major sections related to stock processing:

1. File Handling
File Creation: The method for creating a file to store stock data.
File Reading: Reading the contents of the stock file (e.g., CSV or XML), parsing it into usable data structures.
2. Data Validation and Parsing
Empty File Check: Handling cases where the file is empty or contains invalid data.
Data Parsing: Extracting relevant data from the rows read from the file.
Type Validation: Validating that the data adheres to expected types (e.g., parsing integers and floats from strings).
3. Data Transformation
Map Population: Transforming parsed data into a structured format, typically using maps to reference stock attributes against their respective keys (e.g., siteId, articleVariantId).
Aggregation: Grouping data by relevant keys (e.g., siteId) to calculate totals or prepare for further processing.
4. Business Logic and Rules Application
Filtering Non-Merchandise Articles: Logic to filter out unwanted records based on specific business rules.
Creating Stock Objects: Populating objects representing stock data (e.g., ItemsStock, SiteAggregatedStock) that are necessary for the next steps.
5. Database Operations
Insertions into Audit Tables: Insert properly structured stock data into audit or primary tables for persistence, including error handling in case of database issues.
Updating Status: Updating internal tracking regarding the processing of stock files, such as success or failure statuses.
6. Custom Exception Handling
Throwing Custom Exceptions: Handling scenarios where specific issues arise (e.g., parsing failures, connection issues) by throwing custom exceptions to be handled higher up in the call stack.
7. Email Notifications
Sending Alerts: Sending email notifications when certain criteria are met (e.g., missing site IDs, failed validations).
8. Generating Response JSON
Creating Response Payloads: Building the JSON payload structure that will be sent out, including all necessary fields and objects that represent the stock data.
9. Final Processing and Return
Returning Final JSON: Returning the generated JSON response for further use (e.g., API responses or downstream processing).
Summary
These sections encapsulate the main components and responsibilities involved in the stock processing logic within the PartnerSalesProcessor. Each of these sections will likely interact with one another, as file reading might directly impact validation, which in turn would influence how data is transformed and subsequently inserted into a database. Understanding these sections can help in maintaining or extending the functionality of the stock processing code.
---------------------------------------------------------------

auditing service
email service
seperate service



0----------------------------------

Migration can be done partner by partner
 
1 : Create a blob trigger logic app. Which reads file from the root directory. 
2 :  We will create separate end points for each type in the controller class. May be we can create partner wise controller class.
For Eg: for India we will create separate end points in IndiaController class for stock and price.
3: In logic app we will include switch action for each partner folder and file type, based on the that we will call different API end points for each type and partner.
 
 
Alternate way
 
1 : Create separate logic app for each partner which will listen to the corresponding partner folder.
2 : In logic App we will include Switch action for file type. Based on that we will call different end points for each file type.

1 code base setup
2 Low level design
3 migration of stock process
4 Use JPA Query as much as possible
4 integrate validation
5 Audit the record
7 E2E testing
 
8 Logic app to trigger
 
Performance Testing
 
 
update to add custom throw exception using java exception only
consider file not parsed parser exception ,timeout exception 
 
filenotfound exception

db and azure conn failed 

upload and download testing failed
 ......tech validations exceptions 
 
 
 -----------------------------------------------
 
 
 To add custom exceptions specifically for various failure scenarios in the "stock" section of your code, you can create a series of custom exception classes and throw them in relevant parts of your existing logic. Below is a set of custom exception classes and modifications in the relevant parts of the PartnerSalesProcessor class.

Step 1: Create Custom Exception Classes
First, create a new file for each exception type in your project. Here's how you can define them:

FileNotParsedException.java

package com.mns.intl.pigw.partner.sales.exception;

public class FileNotParsedException extends Exception {
    public FileNotParsedException(String message) {
        super(message);
    }

    public FileNotParsedException(String message, Throwable cause) {
        super(message, cause);
    }
}
TimeoutException.java

package com.mns.intl.pigw.partner.sales.exception;

public class TimeoutException extends Exception {
    public TimeoutException(String message) {
        super(message);
    }

    public TimeoutException(String message, Throwable cause) {
        super(message, cause);
    }
}
FileNotFoundException.java

package com.mns.intl.pigw.partner.sales.exception;

public class MyFileNotFoundException extends Exception {
    public MyFileNotFoundException(String message) {
        super(message);
    }

    public MyFileNotFoundException(String message, Throwable cause) {
        super(message, cause);
    }
}
DatabaseConnectionException.java

package com.mns.intl.pigw.partner.sales.exception;

public class DatabaseConnectionException extends Exception {
    public DatabaseConnectionException(String message) {
        super(message);
    }

    public DatabaseConnectionException(String message, Throwable cause) {
        super(message, cause);
    }
}
AzureConnectionException.java

package com.mns.intl.pigw.partner.sales.exception;

public class AzureConnectionException extends Exception {
    public AzureConnectionException(String message) {
        super(message);
    }

    public AzureConnectionException(String message, Throwable cause) {
        super(message, cause);
    }
}
UploadDownloadTestingException.java

package com.mns.intl.pigw.partner.sales.exception;

public class UploadDownloadTestingException extends Exception {
    public UploadDownloadTestingException(String message) {
        super(message);
    }

    public UploadDownloadTestingException(String message, Throwable cause) {
        super(message, cause);
    }
}
package com.mns.intl.pigw.partner.sales.exception;
 
import org.slf4j.Logger;

import org.slf4j.LoggerFactory;

import org.slf4j.MDC;
 
public class CustomFileNotFoundException extends Exception {

    private static final long serialVersionUID = 1L;

    private static final String EXCEPTION_CODE = "PIGW-PARTNER-FILE-NOT-FOUND-ER009";

    private static final String ERROR_DESCRIPTION = "FILE NOT FOUND";

    private final transient Logger logger = LoggerFactory.getLogger(this.getClass());
 
    public CustomFileNotFoundException() {

        super(ERROR_DESCRIPTION);

    }
 
    public CustomFileNotFoundException(String message) {

        super(message);

    }
 
    public CustomFileNotFoundException(String message, Throwable cause) {

        super(message, cause);

    }
 
    public String getExceptionCode() {

        return EXCEPTION_CODE;

    }
 
    public String getErrorDescription() {

        return ERROR_DESCRIPTION;

    }
 
    public String getErrorMessage() {

        return EXCEPTION_CODE + " : " + ERROR_DESCRIPTION + " : " + getMessage();

    }
 
    public void logErrorMessage() {

        MDC.put("exceptionCode", getExceptionCode());

        MDC.put("errorMessage", getErrorMessage());

        logger.error("CustomFileNotFoundException", this);

        MDC.clear();

    }

}
 this is sample 1
Step 2: Modify the PartnerSalesProcessor Class
Now, you need to update your PartnerSalesProcessor class. Below is an excerpt that shows how to include these exceptions in relevant parts of your logic, especially in the stock handling methods.

Example Modifications

public String responseForStock(String filename, final ExecutionContext context, byte[] content, String partnerCode,
                                List<String> destinationFieldName, List<String[]> allRows) {

    List<String> siteIdList = getPartnerSiteId(partnerCode);
    List<String> skipArticleList = getSkipArticalNo();
    
    Set<String> newSiteIds = new HashSet<>();
    Integer itemCountSize = 0;

    try {
        List<Map<String, String>> fieldObjectList = new ArrayList<>();
        for (int i = 0; i < allRows.size(); i++) {
            String[] data = allRows.get(i);
            Map<String, String> fieldObject = new HashMap<>();
            String check = data[1].substring(0, 1);

            // Filter non merchandise articles
            if (!skipArticleList.contains(check)) {
                fieldObject.put(destinationFieldName.get(0), data[2]); // siteId
                fieldObject.put(destinationFieldName.get(1), data[1]); // articleVariantId
                fieldObject.put(destinationFieldName.get(2), data[0]); // aggregatedStockDate
                fieldObject.put(destinationFieldName.get(3), data[3]); // siteName
                fieldObject.put(destinationFieldName.get(4), data[4]); // storageLocationCode
                fieldObject.put(destinationFieldName.get(6), data[5]); // unrestrictedQuantity
                fieldObject.put(destinationFieldName.get(7), data[6]); // stkInTransQuantity
                fieldObject.put(destinationFieldName.get(8), data[7]); // blockedQuantity
                fieldObjectList.add(fieldObject);

                itemCountSize++;
            }
        }

        logger.info("Counting number of rows:{}", itemCountSize);
        if (fieldObjectList.isEmpty()) {
            throw new FileNotParsedException("No valid data found in the stock file to process.");
        }

        List<SiteAggregatedStock> siteList = new ArrayList<>();
        Map<String, List<Map<String, String>>> purchaseOrderGrpByMap = fieldObjectList.stream()
                .collect(Collectors.groupingBy(x -> x.get(destinationFieldName.get(0))));
        ...
    } catch (SQLException e) {
        throw new DatabaseConnectionException("Database connection failed while processing stock data.", e);
    } catch (URISyntaxException | StorageException e) {
        throw new AzureConnectionException("Azure connection failed.", e);
    } catch (IOException e) {
        throw new MyFileNotFoundException("File not found while processing stock data.", e);
    } catch (Exception e) {
        throw new UploadDownloadTestingException("Upload/Download testing failed.", e);
    }

    return null;
}
Step 3: Implement Exception Handling where Needed
Finally, ensure that you handle these exceptions in your service or controller layer where you're invoking the responseForStock() method. This may involve returning appropriate HTTP status codes or error messages to the client when these exceptions are thrown.

Conclusion
With these changes, your application will have robust error handling for specific scenarios related to file parsing failures, database connection issues, Azure connection issues, and others within the stock section. By throwing custom exceptions, it makes it easier to understand and manage different failure cases in your application.


}
}.......in only stock section of the code.............
update to add custom throw exception using java exception only
consider file not parsed parser exception ,timeout exception

filenotfound exception

db and azure conn failed

upload and download testing failed
......tech validations exceptions

------------------------------------------------------------------

overview of the custom exceptions within the com.mns.intl.pigw.exception package, along with their descriptions:

Custom Exceptions Overview
AzureConnectionException

Description: Thrown when there is an issue connecting to Azure services.
CloudFileDownloadException

Description: Thrown when there is an error downloading a file from the cloud.
CloudFileUploadException

Description: Thrown when there is an error uploading a file to the cloud.
ConfigException

Description: Thrown when there is a problem with the application configuration.
CustomFileNotFoundException

Description: Thrown when a file that is being looked for cannot be found, providing a custom exception alternative to the standard FileNotFoundException.
DatabaseConnectionException

Description: Thrown when there is a failure to connect to the database.
EmptyFileException

Description: Thrown when an operation encounters an empty file where content is expected.
FileNotParsedException

Description: Thrown when a file fails to be parsed as expected.
JSONGenerationFailure

Description: Represents a failure state when generating JSON data; may be a class or a constant indicating an error in JSON generation.
TimeoutException

Description: Thrown when an operation does not complete within a specified time period.




----------------------------------------------------------------


This table outlines the key fields involved in managing partner stock data, highlighting which fields are mandatory (M) or optional (O), their descriptions, and some remarks or questions relating to data usage and interpretation. Here's a brief explanation:

Partner Name (NA): Identifier for the partner supplying the data; not mandatory.
Site Id (M): Unique ID for the partner's site; needed for site-level stock tracking.
Site Name (M): Name corresponding to the Site Id.
Stock Created Date (M): Date when the stock data file was generated (start or end of day).
Article Variant Id (M): Unique product variant identifier.
GTIN / UPC (O): Optional global product codes like EAN or UPC for enhanced identification.
Total Quantity (M): Total stock quantity for the specific product variant.
SubLocationName (M): Code for sub-location within a storage site, showing finer granularity like third party allocation or customer put-away. Questions remain on its exact granularity and whether it updates dynamically through the supply chain.
Storage Location Code (M): Code representing the particular storage location; concerns about whether damaged or written-off stock is included.
Stock Type (M): Classification of stock status like UNRESTRICTED or QUARANTINED.
Unit of Measure (M): Units in which the quantities are expressed (e.g., each).
Quantity (M): Stock quantity count.
Storage Type (O): Whether items are stored boxed or hanging.
First Receipted Date (O): The date stock was first received; optional, may be captured internally rather than from partners.
Additional Commentary:

Partner pricing or selling value is treated separately from stock data and requires a different data feed.
Seasonality for stock can be inferred by linking product master data rather than direct stock data.
Some field details call for further clarification, like the handling of damaged stock or precise meaning of sub-locations.


----------------------------------------------------------


Overview
This data represents a structured message containing inventory and stock information for a partner store/location, typically used in supply chain or retail systems. It is organized into two main groups: Header and Payload, with multiple hierarchical levels to categorize detailed stock data.

Header
Contains metadata about the message itself, including:

MessageId: Unique identifier for the message.
CreatedTimeStamp: Timestamp when the file/message was created.
SourceApplication: The system generating the message (e.g., PIGW).
InterfaceId: Identifier of the interface that transmitted the message.
Version: Schema version of the interface.
InterfaceName: Name/type of the interface.
partnerNumber: Partner’s SAP site ID (business partner identifier).
partnerName: Partner’s name or location.
ItemsCount: Number of records or items included in the message.
Payload
Contains detailed site-specific and stock-related information.

SiteAggregatedStock: Represents aggregated stock data for a specific store or site.
SiteId: Internal site/store reference ID.
SiteName: Name of the partner’s store/location.
ConcatSiteId: Combination of partner code and site ID as a unique reference.
AggregatedStockDate: Date of the stock aggregation.
Items: Detailed product stock data organized hierarchically:
MnSReferences: M&S specific product identification details.
ArticleVariantId: Variant code of the product.
GTIN / UPC: Global Trade Item Numbers and Universal Product Codes.
Quantity: Total quantity of stock available.
Location: Details about product storage locations.
SubLocationCode / SubLocationName: Specific sub-location identifiers, often linked to third parties.
StorageLocationCode / StorageLocationName: Storage area details where stock is held.
StockQuantity: Further breakdown of stock types and quantities.
StockType: Categories like UNRESTRICTED (available stock), INTRANSIT (in movement), BLOCKED (not available).
UnitOfMeasure: Unit in which stock is counted (e.g., Each (EA)).
Quantity: Quantity of stock per stock type.
Storage type: Format of storage such as Boxed or Hanging.
First receipted date: Date when inventory was first received.
Purpose
This structured data format is designed to clearly communicate stock levels, storage details, and product information between systems or partners, supporting tracking, inventory management, and supply chain transparency.


--------------------------------------------------

This table appears to be a mapping of inventory locations for a particular partner or set of partners. Here's a brief explanation of each column and what the table represents:

partnerCode: A unique identifier for a business partner or company.
inventoryLocationCode: A code that identifies a specific inventory location within the partner's system.
inventoryLocationDescription: A textual description of what the inventory location is or its purpose (e.g., Retail, Online, Excess, etc.).
partnerInventoryLocationCode: Another code, possibly standardized or used by the partner or another system, that corresponds to the inventory location.
What the table shows:
For each partner, it links their internal inventory location codes and descriptions to another code system (partnerInventoryLocationCode). This is often used in supply chain management or inventory systems to standardize locations between different partners or systems.

Example:
Partner 9328 has an inventory location with code 1001, described as Online, which corresponds to partner inventory location code 1003.
Partner 9328 also has a location 1002 described as Soil & damage, mapped to code 3995 in the partner's code system.
This mapping helps integrate and synchronize inventory information across different platforms or partners, ensuring accurate tracking and reporting of stock located in various places or assigned specific statuses (like "Return Management" or "Pack and Hold SD").

-------------------------------------------------------

High-Level Design Preparation: First, create a high-level design for how stock data will be sent in a strategic feed.
Discussion with Beam: It's essential to discuss this design with Beam (an internal team or system) to understand their exact requirements. This includes:
The data structure
Additional fields needed (beyond the current data)
Data contract and data mapping specifics
Data Contract: After understanding Beam’s expectations, formalize a data contract that defines what data will be exchanged, in what format, and any validation rules.
Partner Coordination: Once the internal agreement with Beam is settled, evaluate what data or information is required from partners. Have discussions with partners about:
Their capability to provide the requested fields
How they handle specific data like in-transit stock information
Validation and Documentation: Document the final strategic stock finance schema with clear validation rules to ensure data quality and consistency.
Summary: Prepare a design → Discuss with Beam to finalize data contract and structure → Then engage partners for data provision → Document everything with validation rules.


----------------------------------------------


xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx


Field Name	Mandatory or Optional	Description	Remarks	India	Cyprus 1	Comments
Partner Name	NA	Partner Name		partnerName		
Site Id	M	Site ID in Partner System	Get partner sites. If so, then we need site master	siteNo		
Site Name 	M	Name of the Site in Partner System		siteName		
Stock Created Date	M	Date when stock file was generated(SoD or EoD)		aggregatedStockDate		
Article Variant Id	M	Product Variant Id		articleVariantId		
GTIN	O	Product GTIN		EAN - M&S GTIN		
UPC	O	Product UPC		EAN - M&S UPC		
Total quantity	M	Total quantity of stock for the article variant Id		quantity		
SubLocationName	M	Sub location code (Allocated to 3rd parties)	Will specify sub-location name and code	SubLocationCode		"Q: Is this store, warehouse, what level of granularity does this go down to? Does this get updated within the supply chain network?
Sub location - To indicate a classification of the storage location from where the stock gets allocated ie., Allocated to 3rd parties, Customer put away."
Storage Location Code	M	Location code	Will specify location name and code	storageLocationCode		Does this include damaged stock that is written off?
Stock Type	M	Type of Stock	UNRESTRICTED, BLOCKED, IN_TRANSIT, QUARANTINED	stockType		
Unit of measure	M	Unit of measure	EA	unitOfMeasure		
Quantity	M	Quantity of stock	200	quantity		
Storage type 	O	Storage type - Boxed or Hanging	Boxed or Hanging			
First receipted date of the inventory?	O	Product First receipted date of the inventory?				"Highly desirable -
Res: 
Can be captured at data end instead of from partners. Can get from the partners, if they have as part of the one-off initially but not worth getting it as part of daily feed."
						"How can we get the Partner selling value to apply to stock data?
Res: I will check with Arch and come back on this. Syed said that when they initially designed, it was specifically called out to have 2 different feeds (one for stock and one for selling price), as stock value is a different entity. "
						"Need to be able to see season etc for stock data - will this just be done by matching to product master data? 
Res: Can be from multiple season orders, not specific to each season, but season can be inferred from the master data. "
.......Group	Hierarchy 1	Hierarchy 2	Hierarchy 3	Field name	Description	Sample Data
Header				MessageId	Unique Id assigned for the Message	30dd879c-ee2f-11db-8314-0800200c9a66
				CreatedTimeStamp	File created timestamp	2023-02-01T18:26:51.564Z
				SourceApplication	Source Application where file being generated	PIGW
				InterfaceId	Interface Reference Number	I3592
				Version	Reference to schema version of interface	1.0
				InterfaceName	Interface Name reference to number	
				partnerNumber	Partner Number refers to M&S SAP site id	9328
				partnerName	Partner name reference to number	India
				ItemsCount	Number of records count from message	100
Payload	SiteAggregatedStock			SiteId	Siteid is internal reference of Partner store/location	2345
				SiteName	SiteName is internal reference of Partner store/location	India
				ConcatSiteId	Combination of partnerCode and siteId	93282345
				AggregatedStockDate	Stock date	2/1/2023
		Items				
			MnSReferences	This block holds M&S references of product from partner inputs & direct mapping		
				ArticleVariantId	M&S artcicle variant number	22342312001
				GTIN	EAN - M&S GTIN	50560033426001
				UPC	EAN - M&S UPC	1426001
			Quantity	Total stock quantity		
				Quantity	Total stock quantity	24
			Location	This block holds M&S references for location and sub location codes		
				SubLocationCode	Sub location code	1004
				SubLocationName	Sub location name (Allocated to 3rd parties)	AJIO PPMP 
				StorageLocationCode	Storage location code	1000
				StorageLocationName	Storage location name where the stock is stored	Retail
			StockQuantity	This block to provide stock details		
				StockType	Type of stock	UNRESTRICTED, INTRANSIT, BLOCKED
				UnitOfMeasure	Unit of measure	EA
				Quantity	Quantity of stock	12
				Storage type 		Boxed or Hanging
				first receipted date of the inventory		
partnerCode	inventoryLocationCode	inventoryLocationDescription	partnerInventoryLocationCode
8251	1012	Retail	1000
9328	1000	Retail	1000
9328	1001	Online	1003
9328	1002	Soil & damage	3995
9328	1003	Excess	3997
9328	1004	Supplying side resolution from receiving side	2996
9328	1005	AJIO PPMP	1005
9328	1006	Confirmed OR Order	1002
9328	1007	AJIO PPMP Confirmed Orders	1004
9328	1008	Short	3996
9328	1009	Pack and Hold SD	1008
9328	1010	Return Management	1009
9328	1011	Soil & damage (Warehouse)	1006
.....


========================================
doc1=

Existing PartnerStock Schema as provided contains mandatory (M) and optional (O) fields related to stock data.
Includes fields like Partner Name, Site Id, Site Name, Article Variant Id, GTIN, UPC, Quantity, SubLocationName, Storage Location Code, Stock Type, Unit of Measure, Storage Type, etc.
Data is being sourced from partners (India, Cyprus 1) with some internal mappings.
Additional needs and clarifications are required around:
Agree on a clear data contract specifying:
Fields, data types, mandatory/optional status,
Data formats, codes, enumerations (e.g., stockType codes),
Additional fields to be added,
Granularity and update of SubLocation and Storage Location data,
Whether damaged or written-off stock is included,
Capturing selling price or value related data,
Linking stock data with season or product master data,
Validation rules and strategic stock & finance considerations.


Confirm Beam's Data Expectations
What Additional Fields to be added as per Beam requirement from the partner stock data feed?
Additional Fields to be Added
Data Granularity and Hierarchy in the structure
Confirm mandatory vs optional fields per Beam’s processing 
Validation rules Beam wants applied on incoming data (e.g., quantity non-negative, valid site IDs, etc).

Partner Engagement Strategy
If new fields are needed from the partners, confirm partner readiness and timelines.
Determine if partners can provide additional metadata (e.g., selling prices, first receipt dates).
Consider any regulatory or compliance documentation required around data sharing.

Summary
Documented Beam data expectations......Define Fields, data types, mandatory/optional status ...granularity hirarchy....List of additional fields with descriptions............Agreed data granularity and field mapping strucrure document...Agreed data granularity and classification document....Finalized data contract....Partner feedback and confirmation....Validation rule list plan....	Complete schema doc
Discuss with Beam to clarify exact data needs and validation.
Incorporate additional fields as required (e.g., selling price feed, damaged stock flag).
Define Fields, data types, mandatory/optional status ...granularity hirarchy and update frequency for location fields.
Prepare a comprehensive data contract for partner engagement.
Establish validation rules and strategic stock finance schema alignment.

-----------------------------------------------


dynatrace=


Yadav, Jitendra 24 minutes 11 seconds
What is that problem notification right?
Yadav, Jitendra 24 minutes 22 seconds
So you can see this is the.
Yadav, Jitendra 24 minutes 31 seconds
So these are the problem notification. Here you can search for pigw.
Yadav, Jitendra 24 minutes 38 seconds
So you can see and this is it.
Yadav, Jitendra 24 minutes 40 seconds
No, you can see this is AI. Basically something we have created. OK.
Yadav, Jitendra 24 minutes 44 seconds
So this is AI based. This automatically basically monitor everything OK.
Yadav, Jitendra 24 minutes 47 seconds
So for example, if something goes wrong here or there, any service goes down or something, then you'll get alert, OK.
Yadav, Jitendra 24 minutes 53 seconds
So this is Vinod Web page of duty OK.
Yadav, Jitendra 24 minutes 57 seconds
This is on in you know e-mail notification. This is e-mail notification has been created for this pigwsp.
Yadav, Jitendra 25 minutes 2 seconds
This will send the e-mail OK, so if you'll see this is there is a simple rules.
Yadav, Jitendra 25 minutes 8 seconds
OK.
Yadav, Jitendra 25 minutes 8 seconds
So there is a e-mail template and then display name is this and then basically we have configure group e-mail ID here and then yeah you can see basically in subject it will say something and then.
Yadav, Jitendra 25 minutes 20 seconds
And so basically we are saying alerting profile OK.
Yadav, Jitendra 25 minutes 22 seconds
So basically the alerting profile we have configured. Now you have to go in alerting profile and everything we have configured in alerting profile OK.
Yadav, Jitendra 25 minutes 29 seconds
So maybe I'll maybe actually this is unnecessary.
KA
Kumar, Alok
25 minutes 33 seconds25:33
Kumar, Alok 25 minutes 33 seconds
Got it. Jitendra. Yeah, got it.

Yadav, Jitendra
25 minutes 34 seconds25:34
Yadav, Jitendra 25 minutes 34 seconds
I'm taking time, OK for everyone, but.
KA
Kumar, Alok
25 minutes 36 seconds25:36
Kumar, Alok 25 minutes 36 seconds
No, no, this much is information is enough.
Kumar, Alok 25 minutes 39 seconds
We can take it forward from here as much as you. This is fine, yeah.

Yadav, Jitendra
25 minutes 42 seconds25:42
Yadav, Jitendra 25 minutes 42 seconds
So basically this is a profile, OK?
Yadav, Jitendra 25 minutes 45 seconds
So if you open the alerting profile, then things I think things are everything is there. OK, so all the things are there. OK, so only thing is you have to just configure some rule and then things will start working OK.
KA
Kumar, Alok
25 minutes 58 seconds25:58
Kumar, Alok 25 minutes 58 seconds
OK.

Yadav, Jitendra
26 minutes 2 seconds26:02
Yadav, Jitendra 26 minutes 2 seconds
OK.
Yadav, Jitendra 26 minutes 2 seconds
Thank you so far.
Yadav, Jitendra 26 minutes 3 seconds
OK.
Yadav, Jitendra 26 minutes 4 seconds
But you can continue.
Yadav, Jitendra 26 minutes 6 seconds
But let's suggest, OK, so you know, let's plan for a configuration, OK.
PM
Patel, Mustakim
26 minutes 13 seconds26:13
Patel, Mustakim 26 minutes 13 seconds
Yeah, sure, sure, jitend.
KA
Kumar, Alok
26 minutes 15 seconds26:15
Kumar, Alok 26 minutes 15 seconds
Jitendra, just one thing.
Kumar, Alok 26 minutes 16 seconds
Actually your view was a little bit different.
Kumar, Alok 26 minutes 20 seconds
Production one for mine is a little bit difference with lesser. So maybe if some query I will be asking you.
Kumar, Alok 26 minutes 28 seconds
There were some little less steps like classic one you had.
Kumar, Alok 26 minutes 32 seconds
I was not able to see because non prod view I had for the metrics.

Yadav, Jitendra
26 minutes 36 seconds26:36
Yadav, Jitendra 26 minutes 36 seconds
No, no. See, maybe you didn't have a raise access request. OK, so you haven't raised then basically you have to raise.
Yadav, Jitendra 26 minutes 43 seconds
I'll see there is a.
Yadav, Jitendra 26 minutes 45 seconds
You know there is a 180 group.
Yadav, Jitendra 26 minutes 46 seconds
I'll provide you access, then be able to get it open OK.
Yadav, Jitendra 26 minutes 53 seconds
And yeah, please go ahead.

=========================================================

To bind custom exceptions or error events in Dynatrace for monitoring and alerting, you will typically do the following:

Context: Binding Custom Exceptions in Dynatrace for Spring Boot
Dynatrace automatically instruments your Spring Boot app through its OneAgent and captures exceptions thrown by your application, including custom exceptions.

However, if you want to explicitly monitor, tag, or alert on specific custom exceptions, here’s how you can proceed:

Steps to Bind Custom Exceptions for Monitoring in Dynatrace
1. Make sure Dynatrace OneAgent is installed and monitoring your app
Make sure your Spring Boot app is instrumented.
Dynatrace automatically detects exceptions including custom ones thrown by your app.
Verify captured exceptions in the Errors or Service > Service Overview > Errors dashboard in Dynatrace UI.
2. Tag Custom Exceptions in Spring Boot Code (optional)
If you want to enrich exceptions with metadata or specific log entries for Dynatrace to pick up easily:

Add structured logging or additional error attributes.
Example: Using MDC (Mapped Diagnostic Context) to attach info:


import org.slf4j.MDC;

try {
    // code that throws custom exception
} catch (FileNotFoundCustomException ex) {
    MDC.put("exceptionType", "FileNotFoundCustomException");
    throw ex;
} finally {
    MDC.clear();
}
Dynatrace can be configured to pick MDC values for better context.

3. Detect and Filter Custom Exceptions in Dynatrace UI
Go to Analyze > Errors.
Use filters to search by exception class name. For example FileNotFoundCustomException.
Dynatrace auto-groups errors by exception types; your custom exceptions should appear there once thrown.
4. Create an Alert (Problem) for Custom Exception
In Dynatrace, go to Settings > Anomaly detection > Custom events for alerting.
Click Add custom event.
Choose Type: Error event.
Specify a filter like:
Error detection based on Exception class = YourCustomExceptionClassName
Or you can filter by Error message or Service.
Define alert sensitivity and notification settings.
This way, whenever your custom exception occurs, Dynatrace generates an alert.

5. Dynatrace Custom Metrics (optional)
If you want to create a custom metric based on counts of custom exceptions:

Use Dynatrace Custom Events API or log monitoring to parse logs with exception info.
Create calculated metrics based on those logs or error events.
Use those metrics in dashboards or alerting profiles.
Summary (How to Bind Custom Exceptions to Dynatrace Monitoring/Alerting)
Step	Action
1. Instrument Application	Ensure Dynatrace OneAgent is running and monitoring your Spring Boot app
2. Throw & Log Custom Exceptions	Throw exceptions as usual, optionally add MDC or structured logs
3. View Exceptions in Dynatrace UI	Exceptions appear automatically in Errors with your custom exception class names
4. Create Custom Events for Alerting	Filter and create alerts for your custom exception types in Dynatrace
5. (Optional) Create Custom Metrics	Use Custom Events API or log metrics for advanced monitoring
If you want, I can guide you with the exact Dynatrace UI steps or sample custom event JSON for API. Just let me know!

----------------------------------------
To send your custom exceptions as part of Dynatrace problem notifications, you need to make sure that Dynatrace identifies these exceptions as problems and then includes them in its alerting/notification system. Here's how to do this step-by-step:

How to Send Custom Exceptions to Dynatrace Problem Notifications
1. Ensure Custom Exceptions Appear as Errors in Dynatrace
Dynatrace automatically collects unhandled exceptions thrown by your Spring Boot app, including custom exceptions. Just make sure:

Your app is instrumented with Dynatrace OneAgent.
Your custom exceptions are actually thrown and not swallowed silently.
Exception stack traces and messages contain identifiable info.
Verify in Dynatrace UI:

Go to Transactions & services > Your Service > Errors.
Look for errors with your custom exception class names (e.g., FileNotFoundCustomException).
2. Configure Custom Error Detection (if needed)
If your exceptions aren't detected explicitly by Dynatrace or you want to fine-tune detection:

Go to Settings > Anomaly detection > Custom events for alerting.
Create a custom error event:
Filter by error attributes like Exception class name equals your custom exception name.
Save it.
This ensures Dynatrace watches those errors explicitly.

3. Create or Tune Problem Detection Policies
Dynatrace creates Problems automatically based on error spikes or anomalies. Custom errors contribute to this as well.

Under Settings > Problem detection, you can tune sensitivity.
In Service-level problem detection, ensure error rate changes trigger problems.
Let custom exceptions be part of these error counts.
4. Set Up Problem Notification Profiles
Your problems will trigger notifications to configured channels. To link these:

Go to Settings > Alerting > Notification integrations.
Create a notification integration for email, Slack, webhook, etc.
This integration will receive problem notifications generated when Dynatrace detects issues including custom exceptions.
5. Use Custom Field Placeholders in Notification Template (Optional)
You can customize notification content:

Use placeholders like ${ProblemTitle}, ${ImpactedEntity}, ${ProblemDetails}, etc.
The notification will include the root cause that often lists the exception names.
For example, the problem details might say: Root cause: FileNotFoundCustomException in service xyz.
6. Test the Setup
Trigger a test custom exception in your app.
Validate Dynatrace detects the error.
Check that a problem is created.
Check your notification channel to confirm you receive a problem alert mentioning the exception.
Summary: How Exceptions Flow into Dynatrace Problem Notifications
Step	What Happens	Notes
Custom exceptions thrown	Monitored by Dynatrace OneAgent	Exception trace & class sent
Dynatrace detects error	Error event recorded with exception details	Potential spike creates a problem
Problem detected	Dynatrace creates a problem	Includes root cause (exception)
Notification profile active	Sends problem via configured channels	Email, Slack, webhook, etc.
Additional Tips
You don’t have to manually send exceptions to Dynatrace; OneAgent does it automatically.
To highlight your exceptions in problems/alerts, use custom events and problem alerting filters.
You can extend notification content using Dynatrace API or webhooks with customized payloads if needed.
If you'd like, I can help with:

Sample Dynatrace custom event filters for your exceptions.
Creating a sample notification integration (email/webhook).
Sample webhook payload personalization to include exception details.
Just ask!

---------------------------------------------------------------------------------------------------------

Second custom exception

Summary
International_PIGW_Prod_A2047_Alert_ProductExtractor_CloudFileUploadException 

matcher
matchesValue(k8s.deployment.name,"*itl-product-extractor-prod*") 
and matchesPhrase(content,"CLOUD FILE UPLOAD FAILURE")

Title : CLOUD FILE UPLOAD EXCEPTION

event type : custom alert (select)

Allow merge : yes

Set of additional key-value properties
PagerdutyServiceName: PIGWProd
Team: International-PIGW
.....This custom exception configurations.....create similar for FileNotFoundException

https://hso67908.apps.dynatrace.com/ui/apps/dynatrace.classic.settings/ui/settings/builtin:logmonitoring.log-events?gtf=-2h&gf=all&filter=summary%3Afilenot&id=71786c94-4f37-34c8-895b-962ae4a96937

Summary

International_PIGW_Prod_A2047_Alert_ProductExtractor_FileNotFoundException

matcher


matchesValue(k8s.deployment.name,"*itl-product-extractor-prod*")
and matchesPhrase(content,"FILE NOT FOUND EXCEPTION")
Title

FILE NOT FOUND EXCEPTION

Event type

custom alert (select)

Allow merge

yes

Set of additional key-value properties

PagerdutyServiceName: PIGWProd
Team: International-PIGW


Summary:
A unique name for the alert to identify it.
Example: International_PIGW_Prod_A2047_Alert_ProductExtractor_CloudFileUploadException
matcher:
A logical condition or filter to identify relevant log messages or events.
matchesValue(k8s.deployment.name,"*itl-product-extractor-prod*"): This checks if the Kubernetes deployment name matches the pattern itl-product-extractor-prod anywhere in the name.
matchesPhrase(content,"CLOUD FILE UPLOAD FAILURE"): This looks for the phrase "CLOUD FILE UPLOAD FAILURE" in the content/message of the log.
Title:
A short descriptive title you’ll see on the alert, e.g., CLOUD FILE UPLOAD EXCEPTION.
Event type:
Specifies the type of event or alert. Here it's a custom alert.
Allow merge:
Allows multiple occurrences of the event to merge into a single alert to reduce noise.
Additional properties:
Extra metadata to help with routing or categorizing the alert, e.g., which team or PagerDuty service to notify.


did we receive any update related to gibraler access .......based on above stock data ......create partner stock test cases like this Sl No	Test Scenarios	Test Cases	Test Step	Test Data	Expected Result
1	Delta_FoodContent	Files are getting generated successfully 	Trigger IFO Food Data API endpoint for Delta refresh		Food Data should be added in Staging table in sql DB
			Trigger IFO endpoint of food data extract with the respective externalProductID		Data should be added in Food content collection
			Trigger the Orch API(Data Integration) for the file creation		Files should be created in azure storage 
					
					
		Files generation got failed  	IFO Food Data API endpoint gets failed		No Food Data will be added in staging table
			Orch API failure		No data will be added in Food content collection
					File will not be created in azure storage
					
2	Fullrefresh_FoodContent	Files are getting generated successfully 	Trigger IFO Food Data API endpoint for Full refresh		Food Data should be added in staging table in sql DB
			Trigger IFO endpoint of food data extract with the respective productExternalID		Data should be added in Food content collection
			Trigger the Orch API(Data Integration) for the file creation		Files should be created in azure storage 
					
					
		Files generation got failed  	IFO Food Data API endpoint gets failed		No Food Data will be added in staging table
			Orch API failure		No data will be added in Food content collection
					File will not be created in azure storage