A computer hardware system having an image modification platform includes a hardware processor configured to initiate the following executable operations. A digital image is received by the image modification platform. A plurality of objects within the digital image are detected using an object segmentation engine of the image modification platform. The plurality of objects are classified using an object classification engine of the image modification platform and involve associating a stakeholder, respectively, to at least two of the plurality of objects. A knowledge map for the digital image is generated based upon the classifying and the stakeholders using a knowledge map engine. A set of constraints are identified for each of the plurality of objects based upon the knowledge map and the plurality of stakeholders. A modified digital image to be published is generated by an interactive image generation engine using the sets of constraints.

BACKGROUND

The present invention relates to digital image modification, and more specifically, to the modifying a digital image using constraints received from multiple parties.

A digital image is oftentimes comprised of many different types of objects. For example, an image whose subject may be a person standing on a beach could include such objects as an umbrella, a blanket, seabird, water, the beach itself, sandals being worn by the person, a beach hat, sunglasses, the swim suit worn by the person among other objects. Additionally, certain of these objects can have special significance to stakeholders (i.e., manufacturers/sellers or other individuals/organizations having an association therewith) of these objects. In situations where the digital image is being presented in a particular communication channel (e.g., in an advertisement or a website), the stakeholder has an interest in presenting the object in a favorable light.

Current technology exists to modify an already-generated image. However, this technology typically involves a single party making and approving of the changes. Consequently, a need exists for a technology that can take an inputted image and from that inputted image both identify stakeholders associated with that image as well as generate a modified image that meets the requirements of these stakeholders.

SUMMARY

A computer-implemented method within a computer hardware system having an image modification platform disclosed. A digital image is received by the image modification platform. A plurality of objects within the digital image are detected using an object segmentation engine of the image modification platform. The plurality of objects are classified using an object classification engine of the image modification platform and involve associating a stakeholder, respectively, to at least two of the plurality of objects. A knowledge map for the digital image is generated based upon the classifying and the stakeholders using a knowledge map engine. A set of constraints are identified for each of the plurality of objects based upon the knowledge map and the plurality of stakeholders. A modified digital image to be published is generated by an interactive image generation engine using the sets of constraints.

The process can also include the image modification platform being configured to initiate, based upon the knowledge map, a multimodal communication with the plurality of stakeholders. The sets of constraints can be identified based upon information received by the image modification platform during the multimodal communication. At least a portion of the sets of constraints can be automatically generated using the information received by the image modification platform during the multimodal communication, and information provided to the stakeholders during the multimodal communication can be based upon the knowledge map. Additionally, a version of the modified digital image can be provided to the stakeholders. Feedback to the version of the modified digital image can be received from the stakeholders provide during the multimodal communication, and the sets of constraints can be modified based upon the feedback. Another version of the modified digital image can be generated using the modified plurality of constraints, and the sets of constraints can be iteratively modified based upon one or more additional versions of the modified image until a last modified digital image provided to the stakeholders is approved by all of the stakeholders. The last modified digital image is the modified digital image to be published. Also, the knowledge map for the digital image can map relationships between objects within the digital image and includes attributes for each of the objects.

A computer hardware system having an image modification platform includes a hardware processor configured to initiate the following executable operations. A digital image is received by the image modification platform. A plurality of objects within the digital image are detected using an object segmentation engine of the image modification platform. The plurality of objects are classified using an object classification engine of the image modification platform and involve associating a stakeholder, respectively, to at least two of the plurality of objects. A knowledge map for the digital image is generated based upon the classifying and the stakeholders using a knowledge map engine. A set of constraints are identified for each of the plurality of objects based upon the knowledge map and the plurality of stakeholders. A modified digital image to be published is generated by an interactive image generation engine using the sets of constraints.

The computer hardware system can also include the image modification platform being configured to initiate, based upon the knowledge map, a multimodal communication with the plurality of stakeholders. The sets of constraints can be identified based upon information received by the image modification platform during the multimodal communication. At least a portion of the sets of constraints can be automatically generated using the information received by the image modification platform during the multimodal communication, and information provided to the stakeholders during the multimodal communication can be based upon the knowledge map. Additionally, a version of the modified digital image can be provided to the stakeholders. Feedback to the version of the modified digital image can be received from the stakeholders provide during the multimodal communication, and the sets of constraints can be modified based upon the feedback. Another version of the modified digital image can be generated using the modified plurality of constraints, and the sets of constraints can be iteratively modified based upon one or more additional versions of the modified image until a last modified digital image provided to the stakeholders is approved by all of the stakeholders. The last modified digital image is the modified digital image to be published. Also, the knowledge map for the digital image can map relationships between objects within the digital image and includes attributes for each of the objects.

A computer program product is disclosed that comprises a computer readable storage medium having stored therein program code. The program code, which when executed by a computer hardware system including an image modification platform, causes the computer hardware system to perform the following. A digital image is received by the image modification platform. A plurality of objects within the digital image are detected using an object segmentation engine of the image modification platform. The plurality of objects are classified using an object classification engine of the image modification platform and involve associating a stakeholder, respectively, to at least two of the plurality of objects. A knowledge map for the digital image is generated based upon the classifying and the stakeholders using a knowledge map engine. A set of constraints are identified for each of the plurality of objects based upon the knowledge map and the plurality of stakeholders. A modified digital image to be published is generated by an interactive image generation engine using the sets of constraints.

The computer program product can also include the image modification platform being configured to initiate, based upon the knowledge map, a multimodal communication with the plurality of stakeholders. The sets of constraints can be identified based upon information received by the image modification platform during the multimodal communication. At least a portion of the sets of constraints can be automatically generated using the information received by the image modification platform during the multimodal communication, and information provided to the stakeholders during the multimodal communication can be based upon the knowledge map. Additionally, a version of the modified digital image can be provided to the stakeholders. Feedback to the version of the modified digital image can be received from the stakeholders provide during the multimodal communication, and the sets of constraints can be modified based upon the feedback. Another version of the modified digital image can be generated using the modified plurality of constraints, and the sets of constraints can be iteratively modified based upon one or more additional versions of the modified image until a last modified digital image provided to the stakeholders is approved by all of the stakeholders. The last modified digital image is the modified digital image to be published. Also, the knowledge map for the digital image can map relationships between objects within the digital image and includes attributes for each of the objects.

DETAILED DESCRIPTION

Reference is made toFIG.2AandFIG.1, which respectively illustrate a methodology200of generating a modified image140from an initial image110using constraints from a plurality of stakeholders130A,130B, and130C and an architecture100including an image modification platform120in which the methodology is employed according to certain aspects of the present disclosure. In operation, the initial digital image110is received by the image modification platform120. A plurality of objects within the digital image110are detected using an object segmentation engine121of the image modification platform120. The plurality of objects are classified using an object classification engine123of the image modification platform120and involve associating a stakeholder130A,130B,130C, respectively, to at least two of the plurality of objects. A knowledge map300(illustrated inFIG.3) for the digital image110is generated based upon the classifying and the stakeholders130A,130B,130C using a knowledge map engine126. A set of constraints are identified for each of the plurality of objects based upon the knowledge map300and the plurality of stakeholders130A,130B,130C. A modified digital image140to be published is generated by an interactive image generation engine129using the sets of constraints.

More specifically and with reference to the methodology200illustrated inFIG.2, in210, a digital image140to be modified is received by an image modification platform120. In220, an object segmentation engine121of the image modification platform120detects discrete objects within the image110. As used herein, the term “object” refers to discrete, non-person elements of an image110. For example, an image110of a person standing at beach could produce, for example, the following objects: an umbrella, a blanket, seabird, water, beach, sandals being worn by the person, a beach hat, sunglasses, and the swim suit worn by the person. The image modification platform120is not limited as to the particular technology used to implement the object segmentation engine121as many existing technologies so capable exist. However, in certain aspects of the image modification platform120, the object segmentation engine121employs a Mask R-CNN, which is a variant of a deep neural network.

A conventional neural network (CNN) is a type of artificial neural network used in image processing and recognition. A conventional CNN employs a convolutional layer, a pooling layer, and a full connected layer. The convolutional layer abstracts the image100as a feature map. The pooling layer downsamples the feature map via the summarization of the presence of features in portions of the feature map. The fully connected layer connects individual nodes (i.e., “neurons”) in the other layers. A R-CNN (or RCNN) refers to a Region-Based CNN. In this variation, bounding boxes are used in object regions, which can be used to classify multiple image regions of the image110. Mask R-CNN builds upon Faster R-CNN and provides, as outputs, for each candidate object, a class object, a bounding-box offset, and an object mask.

In230, objects identified in220are classified (e.g., labeled) using an object classification engine123of the image modification platform120. Although illustrated as being separate from the object segmentation engine123, a portion or all of the object classification engine123can be incorporated within the object segmentation engine121. The object classification engine123is not only configured to classify each item, the object classification engine123is configured to identify one or more stakeholders130A,130B,130C associated with each object in the image110. For example, assuming that one of the objects identified was shoes, the object classification engine123, using information contained within a data repository125, can identify the particular manufacturer of the particular brand of shoe being illustrated in the image110.

For example, many types of shoes (as with other objects) have visually-distinguishable characteristics (e.g., a logo) that can be used to identify the particular stakeholder. The information contained within the data repository125can store the relationship between these stakeholders130A,130B,130C and their respective visually-distinguishable characteristics of the objects. These associations can be built using the object classification engine123as well as being provided by the stakeholders130A,130B,130C themselves. In this manner, an identity (i.e., classification) of the object along with an associated stakeholder130,130B,130C of the object can be determined.

In240, a knowledge map300(also referred to as a “K-map”) is generated for the particular image110using a K-map engine126of the image modification platform120. An example of a K-map300is illustrated inFIG.3. The generation of a knowledge map300is a well-known technology that has been used in many different contexts. In the particular context of the present invention, a specialized knowledge map300is being used to map relationships between nodes (i.e., objects of the image110) as well as including hierarchical attributes of the nodes. Although not limited in this manner, these relationships between nodes can include relative positioning as well as being part of a common grouping. An example of a common grouping is all the objects (e.g., a shirt, shoes, pants, glasses, and a hat) associated with a particular person within the image110. Although not limited in this manner, attributes of objects can include lighting conditions, stakeholder associated with the object, relative visibility (e.g., clearly visible, obscured, etc.), relative size of object (e.g., 10% of image110), relationships between objects (e.g., a particular shirt is paired with particular pants), and relationships with other stakeholders (e.g., stakeholder A may have a positive (or negative) relationship with stakeholder C). The K-map engine126can draw upon both information generated by the object classification engine123(e.g., the identity of the objects along with an associated stakeholder130,130B,130C of the object) as well as information from the data repository125(e.g., stakeholder relationships and object relationships).

In250, upon generation of the knowledge map300, an interface127of the image modification platform120triggers multimodal electronic communications with stakeholders130A,130B,130C identified by the object classification engine123and found within the knowledge map300. The purpose of these multimodal communications is to identify a set of constraints with regard to a modified image140to be generated from the initial image110. These constraints can pertain to the usage of the modified image140and/or to the generation, by the interactive image generation engine129, of the modified image140. Generation of the modified image140will be discussed in more detail with regard to operation260.

An example of a constraint pertaining to the usage of the modified image140can restrict publishing of the modified image140to one or more particular channels. Although not limited in this manner, a particular channel can be a user's personal website, a website associated within one or more of the stakeholders130A,130B,130C, and/or in a particular marketing context. Examples of constrains pertaining to the generation the modified image140can include but are not limited to positional constraints, visual constraints, privacy constraints, and neighboring constraints. An example of a positional constraint pertains to a relative position of an object within the modified image140. For example, an object that is placed next to another object, which makes it smaller/lesser in comparison. Examples of visual constraints can include whether the distinguishing feature (e.g., a logo) of the stakeholder130A,130B,130C is visible, whether the object is damaged or otherwise depicted poorly (e.g., a soiled shirt), or whether the object is in focus and/or obscured by another object. An example of a privacy constraint can include whether the object should be in the modified image140at all. Another example would be to prevent objects/documents within the image110from including a personal identification. Examples of neighboring constraints regards whether certain objects should (or should not) be placed adjacent one another. For example, a vegan item being placed adjacent a non-vegan item may be prevented.

The interface127is not limited in the manner by which these constraints are determined. In certain aspects, these constraints can be directly provided by the stakeholders130A,130B,130C. Additionally or alternatively, the image modification platform120can automatically generate these constraints. For example, the interface127can an AI-implemented chatbot that can be used to simulate conversation through communication technologies such as messaging applications, websites, mobile apps, etc. by which indications of these constraints can be gathered from the stakeholders130A,130B,130C using, for example, directed questions/questionnaires to which the individual stakeholders130A,130B,130C can respond. The interface127can collect these constraints contemporaneously with the creation of the K-map300. In addition to or alternatively, one or more of the constraints can be retrieved from the data repository125, which were stored after previously being collected (e.g., during the creation of a different modified image). Notably, the K-map300is used by the interface127to identify both the stakeholders130A,130B, and130C to which the constraints apply as well as attributes of an object, which may be the subject of the constraints.

An example of an automatic generation of constraints by the image modification platform120can include parsing feedback210received by the stakeholders130A,130B,130C in response to a proposed version of the modified image140. Prior to the modified image140being published, the interface127can submit the modified image140to the relevant individual stakeholders130A,130B,130C for their respective approval and/or comments via the interface127. Based upon the approval (or lack thereof) and/or comments, the modified image140can be published or the image110can be modified again. Accordingly, the generation of the modified image140and the identification of the constraints can be an iterative process that repeats until an approval has been received from all of the relevant stakeholders130A,130B,130C.

Additionally, the interface127can take the approval and/or comments as feedback210by which to automatically generate/modify the constraints. For example, the interface127can employ natural language processing and a cognitive engine to determine that the feedback comment of “always show watch in turn on mode” requires that visual constraint for the object “watch” requires that the watch be visual depicted in turn on mode. As another example, the interface127can automatically generate a constraint that if a product of Stakeholder A is within the modified image140, then a product of Stakeholder B cannot be within the modified image140based upon a comment of “if P product is in the photo, we do not want to be in the photo.” As yet another example, if a comment was received that “it is too cloudy out,” the interface127may automatically generate a visual constraint that a particular object always be illustrated in a sunny environment.

In260, once a set of constraints has been generated, the modified image140is generated based upon the set of constraints using the interactive image generation engine129of the image modification platform120. Although the interactive image generation engine129is not limited to a particular technology that is used to generate a modified image140, in certain aspects, the interactive image generation engine129employs a modified version of generative adversarial network (GAN) technology, which is a known technology used to generate images. In generating a modified image140, the constraints for a particular object serve to produce three different results. One result is that the object is unchanged. The second result is that the object is removed. The third result is that certain visual/position characteristics of the object are changed (e.g., the focus can be modified, positioning changed, a different view angle, different lighting, etc.).

Referring toFIG.2B, further aspects of an exemplary image generation engine129is disclosed. Using an initial constraint set162, a generator164of the image generation engine129creates a first version of the modified image140. While the generator164of the interactive image generation engine129uses constraints provided by the stakeholders130A,130B,130C, the generator164can also use an internal constraint referred to herein as a popularity score. The popularity score is a score that reflects a computed expected reception for the entire composition of the modified image140. For example, using the constraints provided by the stakeholders130A,130B,130C, the generator164may produce a modified image140that meets all of these constraints, but this modified image140may not be compositionally aesthetically pleasing, which would be reflected in the popularity score. The individual objects within the modified image140, for example, may be out of place relative to their context. Consequently, the generator164can be configured to not only generate a modified image140that meets the constraints provided by the stakeholders130,130B,130C, but also to generate a modified image140that is aesthetically pleasing to a potential viewer of the modified image140.

As discussed above, the interface127of the image modification platform120provides a version of the modified image140to the stakeholders130, which subsequently provide feedback210as this version of the modified image140. As also discussed above, this feedback210can be converted into additional constraints166that can be added to the initial constraint set162. The constraint set162, as modified, is then used to generate another modified image140that can be then sent to the stakeholders130for additional feedback210. This is an iterative process that continues until all of the stakeholders130approve of the last provided version of the modified image140.

In270, once a final modified image140has been generated and approved by all of the relevant stakeholders130A,130B,130C, the final modified image140can be published. As used herein, the term “publish” refers to providing the digital content, e.g., the modified image140, to a particular electronic communication channel. As discussed above, one of the constraints include restricting the publishing of the final modified image140to a particular communication channel.

As defined herein, the term “responsive to” means responding or reacting readily to an action or event. Thus, if a second action is performed “responsive to” a first action, there is a causal relationship between an occurrence of the first action and an occurrence of the second action, and the term “responsive to” indicates such causal relationship.

As defined herein, the term “processor” means at least one hardware circuit (e.g., an integrated circuit) configured to carry out instructions contained in program code. Examples of a processor include, but are not limited to, a central processing unit (CPU), an array processor, a vector processor, a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA), an application specific integrated circuit (ASIC), programmable logic circuitry, and a controller.

As defined herein, the term “server” means a data processing system configured to share services with one or more other data processing systems.

As defined herein, the term “client device” means a data processing system that requests shared services from a server, and with which a user directly interacts. Examples of a client device include, but are not limited to, a workstation, a desktop computer, a computer terminal, a mobile computer, a laptop computer, a netbook computer, a tablet computer, a smart phone, a personal digital assistant, a smart watch, smart glasses, a gaming device, a set-top box, a smart television and the like. Network infrastructure, such as routers, firewalls, switches, access points and the like, are not client devices as the term “client device” is defined herein.

As defined herein, the term “user” means a person (i.e., a human being).

FIG.4is a block diagram illustrating example architecture for a data processing service400for executing the image modification platform120. The data processing system400can include at least one processor405(e.g., a central processing unit) coupled to memory elements410through a system bus415or other suitable circuitry. As such, the data processing system400can store program code within the memory elements410. The processor405can execute the program code accessed from the memory elements410via the system bus415. It should be appreciated that the data processing system400can be implemented in the form of any system including a processor and memory that is capable of performing the functions and/or operations described within this specification. For example, the data processing system400can be implemented as a server, a plurality of communicatively linked servers, a workstation, a desktop computer, a mobile computer, a tablet computer, a laptop computer, a netbook computer, a smart phone, a personal digital assistant, a set-top box, a gaming device, a network appliance, and so on.

The memory elements410can include one or more physical memory devices such as, for example, local memory420and one or more bulk storage devices425. Local memory420refers to random access memory (RAM) or other non-persistent memory device(s) generally used during actual execution of the program code. The bulk storage device(s)425can be implemented as a hard disk drive (HDD), solid state drive (SSD), or other persistent data storage device. The data processing system400also can include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from the local memory420and/or bulk storage device425during execution.

Input/output (I/O) devices such as a display430, a pointing device435and, optionally, a keyboard440can be coupled to the data processing system400. The I/O devices can be coupled to the data processing system400either directly or through intervening I/O controllers. For example, the display430can be coupled to the data processing system400via a graphics processing unit (GPU), which may be a component of the processor405or a discrete device. One or more network adapters445also can be coupled to data processing system400to enable the data processing system400to become coupled to other systems, computer systems, remote printers, and/or remote storage devices through intervening private or public networks. Modems, cable modems, transceivers, and Ethernet cards are examples of different types of network adapters445that can be used with the data processing system400.

As pictured inFIG.4, the memory elements410can store the components of the image modification platform ofFIG.1. Being implemented in the form of executable program code, these components of the data processing system400can be executed by the data processing system300and, as such, can be considered part of the data processing system400.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Referring now toFIG.5, illustrative cloud computing environment550to be used with the FSD assistive system115is depicted. As shown, cloud computing environment550includes one or more cloud computing nodes510with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone554A, desktop computer554B, laptop computer554C, and/or automobile computer system554N may communicate. Nodes510may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment550to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device. It is understood that the types of computing devices554A-N shown inFIG.5are intended to be illustrative only and that computing nodes510and cloud computing environment550can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).

Hardware and software layer660includes hardware and software components. Examples of hardware components include: mainframes661; RISC (Reduced Instruction Set Computer) architecture based servers662; servers663; blade servers664; storage devices665; and networks and networking components666. In some embodiments, software components include network application server software667and database software668.

Virtualization layer670provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers671; virtual storage672; virtual networks673, including virtual private networks; virtual applications and operating systems674; and virtual clients675.

Workloads layer690provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation691; software development and lifecycle management692; virtual classroom education delivery693; data analytics processing694; transaction processing695; and operations of the image modification platform696.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be accomplished as one step, executed concurrently, substantially concurrently, in a partially or wholly temporally overlapping manner, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “includes,” “including,” “comprises,” and/or “comprising,” when used in this disclosure, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Reference throughout this disclosure to “one embodiment,” “an embodiment,” “one arrangement,” “an arrangement,” “one aspect,” “an aspect,” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment described within this disclosure. Thus, appearances of the phrases “one embodiment,” “an embodiment,” “one arrangement,” “an arrangement,” “one aspect,” “an aspect,” and similar language throughout this disclosure may, but do not necessarily, all refer to the same embodiment.